Europe is in a grumpy mood these days. The scale of voter disenchantment with the EU was manifest in May’s European Parliament election results, with Eurosceptic parties topping the polls in France, the UK and Denmark and winning seats in Italy, Germany, Austria, Hungary, the Netherlands, Greece, Sweden and elsewhere.

The mood of pessimism isn’t just a problem for federalists. It carries serious risks for the protection of human rights and the rule of law in Europe.

Since the Lisbon treaty, there has been growing if cautious recognition that the values the Union is supposed to be founded on need to be protected if they are to have meaning. There is now a Charter of Fundamental Rights binding on EU institutions, and on member states when applying EU law. There is a Fundamental Rights Agency. The EU Court of Justice applies human rights in its rulings. A human rights commissioner was added to the European Commission. And the Parliament stepped up its scrutiny of human rights abuse.

These measures are not perfect. The Commission is often timid and reluctant to use infringement proceedings in human rights cases. Refugee pushbacks by Bulgaria are a recent example. The Parliament’s partisan instincts can prevent it from focusing on problems in individual member states. The rights agency remains disconnected from any mechanism to compel political action to remedy the problems it identifies.

The Council has been the most reluctant. The recent Justice and Home Affairs Council endorsement of the idea of a strategy on fundamental rights inside the EU is a positive step. But the Council’s working group on fundamental rights is largely focused on EU accession to the European Convention and pales in comparison to its counterpart on human rights in external relations.

Some may ask why the Council of Europe can’t deal with these tasks? After all, all EU states are members, it is the seat of the European Convention and Court of Human Rights, and has an explicit mandate to protect human rights.

The answer is that while the European Court has some teeth, in general the Council of Europe’s authority is moral rather than political.

The case of Hungary illustrates the limits of its power to tackle rights abuse. With a supermajority in Parliament, Hungary’s ruling Fidesz party used its first term to ram through a new constitution and weaken checks and balances on executive authority, particularly the courts and media. Recent pressure on civil society groups and curbs on the media suggests that it intends to follow a similar path in its second term.

Hungary’s authoritarian slide marshaled a widespread response by the Council of Europe, including multiple reviews by the Venice Commission, its constitutional expert body, condemnation by its Commissioner for Human Rights and statements by its Secretary-General. These were broadly useful interventions but a determined government can ignore such admonitions. What the Council of Europe could not do was deliver political action. An effort to put Hungary under active monitoring by its Parliamentary Assembly failed.

The European Union’s attempts to arrest Hungary’s authoritarian slide were slightly more successful. The Commission took legal action, and the European Court of Justice forced Budapest to back down over the forced early retirement of judges. Unfortunately, the ruling was largely symbolic since Hungary had already replaced a swath of judges. In addition, the Commission’s actions focused on narrow issues, making it easy for the Hungarian authorities to make cosmetic changes while leaving underlying problems unaddressed. The European Parliament produced a strong report on Hungary, but no action followed.

The procedure under article 7 of the EU treaty for risks of a serious breach of EU values would have made sense given the structural issues.  Used carefully, as human rights groups and some in the Parliament advocated, it could have persuaded Hungary to change course. But there was no appetite in the Council or most of the Commission for a procedure that could have ultimately led to suspension of Hungary’s voting right

Recognition of the Council’s weak response to the crisis in Hungary, led the foreign ministers of Germany, the Netherlands, Denmark and Finland in March 2013 to call for more vision in the EU’s protection of human rights at home. The Commission’s response, a communication on strengthening the rule of law, is a modest but positive step that could help in a crisis, though it leaves unaddressed the chronic human rights abuses across member states.

Prior to the Parliament elections, the idea of stronger EU rights protection for those inside its borders had gathered some momentum. The worry is that the election results will derail it. If EU member states conclude that voters want less EU action they may step back from the effort, the idea of a strategy notwithstanding. In fact there are already signs that the Commission’s communication lacks full support from member states. The Commission may also be reluctant to act boldly for fear of further backlash. And the Parliament, even though a majority of MPs still represent four mainstream political groups, may also shy away from supporting a positive EU vision of human rights protection.

That would be a mistake. Giving the EU a stronger role in protecting human rights could actually cement public support for the Union and the values it embodies. Doing so would require effective communication and leadership from EU governments, and commitment from EU institutions. But a Union that actually protects the rights of those living inside it would be a powerful rejoinder to those who assert that Europe is a bankrupt idea.

Benjamin Ward is deputy director of the Europe and Central Asia Division at Human Rights Watch.