Tom Roche

• Does it make sense to talk about Org Mode instead of Markdown?
• The export problems
• Some examples to make it clearer
• The solution: export from Org-Mode to Markdown via Pandoc with GFM option.
• The special case of exporting code blocks.

Does it make sense to talk about Org Mode instead of Markdown?

Why should I use Org-Mode to write articles to be published network in Markdown format?

Wouldn’t it be more convenient to write the articles directly in Markdown?

The question is more than justified and the answer is not obvious.

First, let me remind you that Org-Mode is a mode of GNU/Emacs. While there are various plugins for other editors, including Vim, none of them can match the outstanding efficiency of the original system.

So: to write in Org-Mode, it’s best to must use GNU/Emacs.

Org-Mode is a complete and extremely powerful structured writing system, although the learning curve is significantly upward.

It is a very useful resource for anyone who has to write complex documents, such as legal documents, scientific publications, novels.

Markdown is a language with text-format syntax much easier and immediate to learn.

However, some current CMSs directly accept Markdown format without the need for any conversion. I am not aware, however, that it is possible to publish directly in Org-Mode format.

Org-Mode (in Emacs, of course) has a very extensive set of commands for moving, shifting, deleting chapters, and managing images and is, therefore, extremely useful in the creative and construction phase of the writings.

In summary: Org-Mode is a very powerful, fast and effective writing system for all those who write complex documents by profession or passion.

But sooner or later, you will need to convert the material from Org-Mode to another format for printing (LaTeX) or web publishing (Markdown or HTML) (or both).

The export problems

Org-Mode has excellent and very fast built-in export capabilities with a myriad of options and configurations to various formats including LaTeX, HTML, and Markdown itself.

The export result is excellent for LaTeX, HTML, and basic Markdown.

However, basic Markdown lacks some syntactic features compared to the GFM dialect (i.e., “GitHub Flavored Markdown”).

For example, as I witnessed directly in the course of preparing one of my articles, Org-Mode’s internal export produces tables in HTML format (i.e., with the typical <table>, <tr> and <td> tags) instead of the dedicated Markdown GFM table markup code.

Using Pandoc with the standard configuration (such as “Pandoc input_file -or output_file”) the result is better but still not enough, as we will see in the examples below.

Some examples to make it clearer

I report some export examples from the Org-Mode version of this article of mine.

The following table is the “source” text version in Org-Mode:

| Function | Keystrokes |
|--------------------------------+------------|
| | <c> |
| Delete Surrounding Command | =dsc= |
| Delete Surrounding Environment | =dse= |
| Change Surrounding Command | =csc= |
| Change Surrounding Environment | =cse= |

This is the result of exporting via Org-Mode built-in function:

 <table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">


<colgroup>
<col class="org-left" />

<col class="org-center" />
</colgroup>
<thead>
<tr>
<th scope="col" class="org-left">Function</th>
<th scope="col" class="org-center">Keystrokes</th>
</tr>
</thead>
<tbody>
<tr>
<td class="org-left">Delete Surrounding Command</td>
<td class="org-center"><code>dsc</code></td>
</tr>

<tr>
<td class="org-left">Delete Surrounding Environment</td>
<td class="org-center"><code>dse</code></td>
</tr>

<tr>
<td class="org-left">Change Surrounding Command</td>
<td class="org-center"><code>csc</code></td>
</tr>

<tr>
<td class="org-left">Change Surrounding Environment</td>
<td class="org-center"><code>cse</code></td>
</tr>
</tbody>
</table>

The code is pure HTML and not Markdown.

This, on the other hand, is the result of exporting with Pandoc using the following formula:

pandoc -s nomefile.org -o nomefile.md

Function Keystrokes
-------------------------------- ------------------
Delete Surrounding Command `dsc`{.verbatim}
Delete Surrounding Environment `dse`{.verbatim}
Change Surrounding Command `csc`{.verbatim}
Change Surrounding Environment `cse`{.verbatim}

The code seems to be correct but the result on browser is very different from the desired one:

 Function Keystrokes

Delete Surrounding Command dsc{.verbatim} Delete Surrounding Environment dse{.verbatim} Change Surrounding Command csc{.verbatim} Change Surrounding Environment cse{.verbatim}

The solution: export from Org-Mode to Markdown via Pandoc with GFM option.

The solution is to convert from Org-Mode to Markdown using Pandoc but adding the GFM option to have the code “enriched” from the base version.

This is the pattern, found, after long search, on this page:

pandoc --from=org --to=gfm org-mode-file.org > markdown.md

where, of course, the document names org-mode-file.org and markdown.md must be replaced with the real ones.

Note the “--to=gfm” option that determines the output in the Markdown GitHub format.

Applying that pattern to the code mentioned in the previous paragraph produces the following converted code:

| Function | Keystrokes |
|--------------------------------|:----------:|
| Delete Surrounding Command | `dsc` |
| Delete Surrounding Environment | `dse` |
| Change Surrounding Command | `csc` |
| Change Surrounding Environment | `cse` |

In this way, you get perfectly formatted Markdown code even for the parts not provided in the basic version.

Note, in addition to the complete table of each character, also the “centering” of the second column.

Then, you can create the writings by taking advantage of the infinite potential of Org-Mode and, then, exporting them to HTML or Markdown (or LaTeX) perfectly formatted for the publishing stage.

The special case of exporting code blocks.

While testing this topic I found a peculiarity in the export of source code (tag src).

Simply using the combination “begin_src ... end_src” gives a result in Markdown as if it were the “quote” format, that is, with four indent spaces (which is equivalent to prefixing “>” before each line).

In exporting via Pandoc, the same result is obtained ONLY if the fragment language is not specified.

If, however, the language is specified (for example: “#+begin_src markdown”) the export is correctly formatted as a code block with the same language specified in Markdown.

The names of the languages allowed by Org and Markdown are not, however, coincident.

This, however, does not preclude the correct formatting of the code when exporting, but only the colored display of the content: if the language indicated in Org-Mode were not provided in Markdown it would simply result in an uncolored fragment but, in any case, correctly exported as a “code block” and not as a “citation block.”

Thank you for your attention.

I'M SO SORRY ABOUT THE AUDIO QUALITY, our good mics over at Corner Späti Heavy Industries GmbH decided to be colonialist and CANCEL our conversation on Irish, Catalan and minority languages in general but we managed to put it out. Take that English Chauvinism, the profit motive and the French conception of state building (and whoever else is responsible for the decline of languages).

the title means "Irish, Catalan and other minority languages"

WHERE TO FIND OUR GUEST:
https://twitter.com/iarphunc
https://rupture.ie/
the article we're referring to throughout the episode is "Tá Gá le Gníomh" written by our wonderful guest, it's available in Issue 8 of Rupture Magazine and will be made available for free on the website at some point soon.

HOW TO SUPPORT US:
https://www.patreon.com/cornerspaeti

HOW TO REACH US:
Corner Späti https://twitter.com/cornerspaeti
Julia https://twitter.com/KMarxiana
Rob https://twitter.com/leninkraft
Nick https://twitter.com/sternburgpapi
Uma https://twitter.com/umawrnkl
Ciarán https://twitter.com/CiaranDold

Support Corner Späti

Our good friends Liz and Brace of TrueAnon stop by to help us send off The Year of the Smile. We discuss their mini-series on Synanon, a substance rehabilitation program turned violent cult, and how it relates to all sorts of American phenomenon: from 1950’s acid tests to the modern Troubled Teen Industry. We also discuss the state of our collapsing tech/media infrastructure, the Epstein case, and why the government should begin issuing quests to citizens.

Check out TrueAnon’s excellent miniseries The Game starting here: https://soundcloud.com/trueanonpod/the-game-part-1-dopefiend,

and the rest on Patreon: https://www.patreon.com/TrueAnonPod

Tickets for the Hell on Earth launch show/party @ Littlefield in NYC 1/20/23 here: https://littlefieldnyc.com/event/?wfea_eb_id=479703214227

Hosted on Acast. See acast.com/privacy for more information.

Click here for the full interview with Luhansk Deputy Foreign Minister Anna Soroka.

War in Ukraine did not start with Russia’s February 2022 invasion.

Ukraine has been at civil war since 2014, when the US backed a coup to overthrow President Viktor Yanukovych. When the new post-coup government cracked down on ethnic-Russian Ukrainians and their Russian culture, Ukrainians in the eastern Donbas region rose up in rebellion, leading to the creation of the Donetsk and Luhansk People’s Republics.

Anna Soroka, joining the Useful Idiots from Ukraine, is the former Deputy Foreign Minister of the Luhansk People's Republic. She joins Useful Idiots to share what it’s like being on the other side of a civil war that has now escalated into a full-blown proxy war with Russia.

Sometimes discussions about war can become abstract, but it’s important to go “behind enemy lines” and hear from real people on the other side.

It’s a perspective that you don’t get to hear at all in the NATO states. Unless you follow shows like Useful Idiots.

Plus, watch our Thursday Throwdown where Biden completes his transformation into Trump as he kills Bernie's bid to stop the Saudi war on Yemen.

It’s all this, and more, on this week’s episode of Useful Idiots. Check it out.

Weeks before he was murdered, Victor Hugo Orcasita presented his wife with a letter describing his last wishes.

Orcasita, a union leader, had been pushing for better conditions at his workplace, a mine in northern Colombia owned by a subsidiary of the Alabama-based coal company Drummond. Then the death threats started coming in. He believed that the armed strangers who had started appearing around the mine’s cafeteria would soon make those threats a reality.

“He foresaw his death,” said his widow, Elisa Almarales Viloria.

On March 12, 2001, paramilitary gunmen dragged Orcasita and another union leader, Valmore Locarno, from a company bus as the men returned home from work. The gunmen shot Locarno on the spot and carried Orcasita off in the bed of their pickup truck. His body was found the next day. He’d been shot in the head, his teeth knocked out.

The miners’ union was convinced that Drummond was involved in the murders. They suspected that the company was secretly paying the paramilitary group that executed their leaders. Ultimately, a Drummond food service contractor who ran the mine’s cafeteria was convicted of plotting the murders and sentenced to 38 years in prison.

To make the case that the company was complicit in the killings, the union turned to Terry Collingsworth, a lifelong human rights attorney based in Washington, D.C.

Victims suing multinational corporations for alleged crimes committed abroad face steep odds. Collingsworth has made a specialty of these uphill battles, devoting his career to holding companies accountable in American courts for human rights abuses overseas. In his struggle with Drummond, he collaborated with activist groups, spoke out in the media, and wrote letters to Drummond’s business partners accusing the company of “hiring, contracting with, and directing” the paramilitaries who committed the murders.

Collingsworth’s decision to file suit in the United States made Orcasita’s widow hopeful that justice would prevail. For years, she had felt that justice would be impossible in Colombia due to Drummond’s political clout.

“What we were most excited about was bringing the lawsuit in Alabama,” she said. “There it would not be so easy for them to traffic their influence.”

Collingsworth lost an initial trial in 2007, when a jury found there wasn’t clear evidence tying the company to the crimes. Another of his lawsuits was dismissed for being too similar to the first. But Collingsworth continued to press his case, offering new witnesses with firsthand testimony implicating Drummond.

Then, in March 2015, the case took a surprising turn.

Drummond had returned fire in the legal fight with an unusual accusation. The company charged that Collingsworth — an advocate who recently brought a case before the U.S. Supreme Court — had led a “multifaceted criminal campaign” to extort Drummond into paying a costly settlement. This campaign, Drummond alleged, was in fact a racketeering conspiracy as defined by the Racketeer Influenced and Corrupt Organizations Act, better known as RICO.

Drummond’s charges represent a scorched-earth legal strategy in which corporations are turning the tables on attorneys and advocates who accuse them of wrongdoing. The technique was popularized by the elite corporate law firm Gibson, Dunn & Crutcher, whose clients include a who’s who of America’s most powerful companies. Representing the oil giant Chevron, Gibson Dunn convinced a judge to block one of the largest environmental verdicts ever reached by deploying a novel formula: using the civil provisions of RICO to charge opposing attorneys with racketeering.

Companies that have used RICO against their accusers say they brought the charges on themselves by committing fraud, bribery, and extortion. In Chevron’s case against environmental attorney Steven Donziger, a federal judge agreed; in the case against Collingsworth, a judge ruled that there was enough evidence of malfeasance to allow discovery. Human rights and environmental advocates contend that the true purpose of the cases is to send attorneys and activists a message: Going toe-to-toe with heavyweight corporations can lead to personal ruin.

“Companies with functionally limitless resources can come in and bigfoot like this, and no one can withstand it.”

Legal experts say some plaintiff’s attorneys made themselves vulnerable to RICO claims because they operated at the most aggressive edge of their field, overstepped ethical lines, and by their own admission made mistakes. By shifting the spotlight to these attorneys’ conduct, corporations effectively sidestepped the original allegations against them. Following these victories, other companies adopted similar theories to target advocacy groups directly.

If the goal is to hold attorneys accountable for unethical behavior, RICO is an odd choice. George Washington University law professor and international human rights attorney Ralph Steinhardt noted that RICO is a “very heavy club to swing” when there are more direct penalties, like sanctions, which punish the advocate without invalidating the entire case.

“One wonders why you would bring out the big guns of racketeering to send a message,” he said. “It’s a take-no-prisoners approach that’s intended to distract from whatever good faith allegations there may be.”

Ken White, a former federal prosecutor who specializes in First Amendment law, said responding to alleged misconduct by opposing attorneys with RICO charges is “like going after raccoons knocking over your trash cans with a tactical nuke.”

What’s missing, White says, is a universal mechanism to secure quick dismissals of baseless RICO claims. “Companies with functionally limitless resources can come in and bigfoot like this, and no one can withstand it,” White said.

Climate activists gather outside the Gibson Dunn office in New York City to protest against Chevron on June 10, 2021.

Photo: Tayfun Coskun/Getty Images

The RICO Playbook

As scientists issue dire warnings about climate change, advocates have turned to the courts and public campaigning to try to impose consequences on companies they accuse of serious attacks on the environment. Energy and extractive industry giants targeted by these efforts have been particularly eager to turn the tables by deploying this no-holds-barred strategy.

One of the world’s biggest oil companies, accused of dumping billions of gallons of toxic waste in the Amazon rainforest, won the first high-profile victory that relied on this approach. Drummond filed RICO charges in response to allegations that it financed the murder of union leaders who threatened the productivity of its coal mines. A pulp and paper company accused of destroying forests and the energy company behind the Dakota Access pipeline followed soon after, bringing RICO claims against environmental campaigners and anti-pipeline protesters.

In each of these cases, the accused racketeers were environmental and human rights attorneys, Greenpeace and other environmental groups, or Indigenous land and water rights activists.

The RICO Act, originally passed in 1970 to help prosecutors go after the mafia, includes civil provisions that allow private parties to allege a racketeering conspiracy. Most civil RICO claims are filed in business disputes, while others have been brought against political groups from anti-abortion protesters to animal rights activists. These suits require a high bar of evidence: They must prove a pattern of at least two “predicate” crimes such as bribery, fraud, or money laundering; that the perpetrators worked together in a criminal “enterprise”; and that the perpetrators acted with criminal intent.

Nonetheless, RICO claims offer powerful incentives to plaintiffs. If a judge allows the case to go forward, the defendants are subject to extensive discovery in which a well-funded corporate law firm can bury them in paperwork. If the company wins and can establish damages, those damages are automatically tripled.

“When we really think about what these suits are about, it’s fear.”

The success of early cases has helped build a body of law that opens the door for even more aggressive uses of the statute. The most recent corporate RICO cases have sought to define common public advocacy techniques such as negative media campaigns that allegedly contained false claims as predicate offenses for racketeering. The financial and reputational costs of defending these claims can make them devastating to their targets even if they ultimately fail.

“These RICO cases are easier to file than they are to win,” Steinhardt said. “Their intimidating purpose is served by their filing or their pendency.”

Deepa Padmanabha, deputy general counsel for Greenpeace USA, said that even though her team was awarded more than $800,000 in legal fees after successfully defeating RICO claims, the cost of defending the case was even higher.

Padmanabha said that two RICO suits would have cost the organization a total of more than a billion dollars if it had lost. The goal of the charges, she believes, was to caution the environmental movement that even the largest organizations were not safe from ruin.

“When we really think about what these suits are about, it’s fear,” Padmanabha said.

Corporate lawyers seem to be betting that the strategy will have staying power. In October 2020, Gibson Dunn announced a new practice in Judgment and Arbitral Award Enforcement, offering its services to creditors or debtors seeking to litigate existing judgments. The practice’s website highlights “its representation of Chevron Corporation in its successful RICO suit” and boasts that the firm “excels at defending companies and individuals against fraudulent arbitration awards and foreign judgments.”

Evan Mascagni, policy director for the Public Participation Project, an organization that fights against abusive lawsuits, said the RICO strategy threatens to overwhelm the legal system by allowing deep-pocketed companies to deploy endless resources to silence critics and defy judgments against them.

“I think if we accept this as a society, as a country, we’re saying we’re going to give incredibly powerful multinational corporations the ability to hijack our legal system,” Mascagni said.

Steven Donziger speaks during a press conference on March 19, 2014, in Quito, Ecuador.

Photo: Rodrigo Buendia/AFP via Getty Images

A Victory for Chevron

The RICO strategy was most famously deployed in 2011 by Chevron in its bitter legal conflict with attorney Steven Donziger.

At the time, Donziger was the lead lawyer pursuing massive damages against the oil company for toxic pollution in the Ecuadorian Amazon. Chevron inherited the lawsuit when it acquired Texaco, which had allegedly left hundreds of open pits of sludge in the rainforests where it operated, causing cancer deaths, miscarriages, and birth defects among the area’s mostly Indigenous residents. As the prospects of a multibillion-dollar judgment grew higher, Chevron enlisted the help of Gibson Dunn.

In February 2011, Gibson Dunn attorneys filed a civil RICO suit in New York accusing Donziger and his colleagues of running a racketeering conspiracy. They charged that Donziger and his team secretly controlled a key independent expert appointed by the Ecuadorian court to assess pollution damages. By the time of Donziger’s trial, they added the accusation that Donziger had bribed an Ecuadorian judge to allow his team to ghostwrite the judgment against Chevron.

Chevron provided hundreds of thousands of dollars in benefits to Alberto Guerra, the witness who claimed he’d facilitated the bribery and served as a liaison between Donziger’s team and the Ecuadorian judge. The benefits included relocating Guerra and his family from Ecuador to the United States, where the company supplied him with a $12,000 monthly salary. Chevron has said that it relocated Guerra to ensure his safety and that the payments were to compensate him for the cost of providing his evidence.

The company’s case was bolstered by Donziger’s own words, obtained through discovery of materials that included outtakes from a documentary film. In one clip, Donziger discussed the size of a possible judgment against Chevron and speculated that his team could “jack this thing up to $30 billion.” In draft testimony in 2013, Donziger conceded that he “did make errors along the way” but challenged the legitimacy of the proceedings against him.

As the RICO case headed for trial, Chevron made a strategic move. Roughly two weeks before the trial date, it dropped its request for damages and sought only to block enforcement of Ecuador’s $9.5 billion judgment. That meant the case would no longer be heard by a jury but decided solely by Judge Lewis Kaplan, a federal district judge in Manhattan who had ruled in the company’s favor in earlier motions.

In March 2014, Kaplan ruled in favor of Chevron, barring U.S. enforcement of the Ecuadorian judgment and holding that private parties are entitled to seek relief from foreign courts’ decisions under civil RICO — a crucial green light for the strategy that Gibson Dunn had developed.

Kaplan concluded that Donziger’s team had not only secretly written the Ecuadorian court’s ruling, but also submitted false evidence and made hidden payments to the court-appointed expert. “The wrongful actions of Donziger and his Ecuadorian legal team would be offensive to the laws of any nation that aspires to the rule of law,” Kaplan wrote in his opinion.

Critics have raised questions about irregularities in the case against Donziger. Guerra later changed key details in his testimony, including the nature of the alleged bribe agreement and the dates of trips in which he claimed to have worked on the case. Computer analysis also showed the judge in question had a running draft of the judgment saved on his hard drive for months, undermining the ghostwriting claim. Still, the case set in motion a stunning downfall for Donziger. The one-time star of the environmental bar ended up serving time in federal prison on contempt charges stemming from his refusal to comply with orders from Kaplan after the RICO decision. Meanwhile, Chevron avoided paying the multibillion-dollar judgment for the toxic sludge that remains in the Ecuadorian Amazon.

In an emailed statement, Gibson Dunn noted that an arbitration panel established through a trade agreement between the United States and Ecuador found that Texaco, Chevron’s predecessor, had complied with a pollution remediation plan approved by the Ecuadorian government, releasing the company from liability. Critics contend that the remediation plan failed to clean up the damage and did not cover claims by private plaintiffs.

In response to questions about Guerra, the firm said Donziger exaggerated the importance of his testimony and pointed to Kaplan’s statement that he would have “reached precisely the same result in this case even without the testimony of Alberto Guerra.” Gibson Dunn added that Kaplan’s RICO ruling, which was unanimously affirmed by a panel of judges on the 2nd U.S. Circuit Court of Appeals, showed that the firm’s advocacy had uncovered serious wrongdoing.

“As for Gibson Dunn’s work successfully exposing fraud by unscrupulous lawyers like Mr. Donziger who seek to rip off vulnerable people in weak legal systems overseas based on lies, this is laudable work vindicating the rule of law,” William Thomson, a partner at Gibson Dunn who was part of its Chevron team, wrote in the statement.

Donziger maintained that his contacts with the Ecuadorian expert were legal and appropriate under Ecuadorian law, and that the ghostwriting charges were fabricated.

“Chevron used a civil racketeering case and false witness testimony from a person who is an admitted liar to try to criminalize me,” Donziger told The Intercept and Type in a written statement. “They wanted to use this bogus RICO case to try to get people to forget about the human devastation Chevron caused in Ecuador.”

An open-air coal mine in Colombia’s La Guajira Department.

Photo: Jeffrey Tanenhaus

Witnesses in Dispute

About a year after Kaplan blocked the Ecuadorian judgment against Chevron, Drummond filed RICO charges against Collingsworth.

Although the company had already prevailed against several of his lawsuits, Collingsworth forged ahead with new legal actions, adding witnesses who offered firsthand testimony alleging that the coal company was complicit in the union leaders’ murders.

One of these witnesses was an imprisoned former paramilitary commander called El Tigre, or the Tiger, who testified that Drummond provided regular payments to his unit. Another key witness was Jaime Blanco, the food contractor who was ultimately convicted of the murders, who said Drummond used his company as a conduit to funnel money to the paramilitaries and directed them to commit the murders.

Collingsworth made payments to El Tigre’s family members and helped arrange financing for Blanco’s legal defense when he agreed to testify. He said the funds he provided to El Tigre’s family were security payments to help the family relocate in order to avoid violent retaliation by the paramilitaries, Autodefensas Unidas de Colombia, which the U.S. State Department designated as a terrorist group in 2001. In response to a court order, Collingsworth disclosed similar payments to relatives of three ex-paramilitary witnesses, but he failed to include the payments to El Tigre and two other ex-paramilitaries, as well as his arrangement with Blanco.

Drummond’s media office did not respond to multiple phone calls and emails requesting comment for this story, and attorneys for Drummond declined to comment.

Colombian authorities have backed up key elements of Collingsworth and El Tigre’s account. In December 2020, the Colombian Attorney General’s Office charged the current and former presidents of Drummond’s Colombian subsidiary with conspiracy in the union leaders’ murders. The 149-page charging document included a summary of a forensic analysis that found evidence of more than $3.7 million in overpayments from the subsidiary to Blanco’s company, bolstering allegations that Drummond had financed the paramilitaries.

Prosecutors also noted that numerous witnesses who did not receive security payments had testified to the same facts. The accounts of El Tigre and other disputed witnesses, they wrote, were “in harmony with and verified by other forms of proof.”

This fall, prosecutors named Drummond’s Colombian subsidiary as a “civilly responsible third party” in the case of the union leaders’ murders.

Though its Colombian subsidiary is now in the crosshairs of prosecutors, Drummond has had more success against Collingsworth in the United States.

In 2015, Drummond filed a civil RICO suit charging that Collingsworth had bribed El Tigre, Blanco, and other witnesses to falsely testify that Drummond was involved in the murders, as part of a racketeering conspiracy to strong-arm the coal producer into paying a hefty settlement. The company pointed to inconsistencies in their testimonies, noting previous statements in which they denied that Drummond had worked with the paramilitaries before they became witnesses for Collingsworth.

The case, which focused on the undisclosed payments to witnesses, was heard by a federal judge who had ruled in Drummond’s favor in earlier litigation with Collingsworth, Judge R. David Proctor of the Northern District of Alabama.

Collingsworth said in court filings that the omissions were an “inadvertent disclosure error” resulting from miscommunication with his co-counsel in Colombia. He said he had failed to include the payments in an initial disclosure and then recycled his answer repeatedly before realizing his error. He also apologized to the judge for making a “terrible mistake” in not revealing his arrangement with Blanco, which he had previously deemed to be outside the scope of required disclosures.

“Sitting here now, boy, I wish I had just disclosed it,” Collingsworth said in a phone interview. “Because it wasn’t hiding the truth or changing the testimony.”

The real question, Collingsworth said, is whether the payments to the witnesses in Colombia were ethical and necessary for their safety. The security arrangements were needed for them to testify truthfully without endangering their families, he said, noting that he reviewed all arrangements in advance with ethics lawyers and turned down witnesses who sought to exchange testimony for cash. He fiercely defends his decision to help relocate the families of former paramilitaries and submitted testimony supporting the need for security payments by expert witnesses including Javier Peña, the former Drug Enforcement Administration agent who led the mission that killed cartel leader Pablo Escobar and inspired the Netflix series “Narcos.”

“It was morally necessary to protect these families from one of the most brutal groups that roamed the earth,” Collingsworth said.

In December 2015, Proctor ruled that Drummond’s RICO case could go forward, finding that Collingsworth’s explanation for the undisclosed payments was “as weak as it is incredible.” He held that there was probable cause to believe that Collingsworth had bribed witnesses and suborned perjury, opening the door to the extensive discovery process that Chevron had effectively used against Donziger.

It was the beginning of years of legal wrangling that Collingsworth said drained the resources of his small human rights firm.

Collingsworth said he has spent some 2,000 hours — what a lawyer usually bills in a year — defending against Drummond’s charges. Even more damaging, he said, has been the impact on his professional reputation, which he says has deprived him of business opportunities and revenue.

“I have had colleagues who are in law firms tell me that they can’t collaborate with me until these charges are completely resolved in my favor, because they don’t want to be accused of associating with someone who bribes witnesses,” Collingsworth said.

Steinhardt, the human rights law professor, said the facts of the case aren’t black and white, but the charges against Collingsworth are disproportionate. “He isn’t a racketeer,” Steinhardt said.

A protester holds a poster outside the constitutional court in Quito, Ecuador, on Nov. 9, 2016, at a demonstration marking 23 of the legal battle over Texaco’s pollution.

Photo: Dolores Ochoa/AP

A Chilling Effect

The success of these cases paved the way for increasingly aggressive uses of civil RICO.

Around 2012, Greenpeace and other environmental groups launched a protest campaign against Resolute Forest Products, accusing the forestry company of destroying boreal forests in Canada. Several years later, Greenpeace and others began another campaign targeting Energy Transfer Partners (now part of Energy Transfer LP), the company behind the Dakota Access pipeline. This campaign charged, among other things, that the company was threatening Indigenous communities’ water supply and sacred sites. Greenpeace and its allies rallied their members, drove media coverage, and urged the companies’ business partners to sever ties unless the companies changed course.

The two companies filed RICO charges against Greenpeace and the other groups in 2016 and 2017. Both were represented by the firm Kasowitz Benson Torres, whose founding partner Marc Kasowitz was a longtime personal attorney for Donald Trump and filed a defamation case against one of Trump’s critics. (First Look Institute, the nonprofit that publishes The Intercept, is involved in litigation with Energy Transfer, represented by the Kasowitz firm, over records related to the Dakota Access pipeline.)

Michael Bowe, the former Kasowitz partner who brought the RICO cases, told Bloomberg in August 2017 that he was in contact with other companies considering similar actions and “would be shocked if there are not many more.” He anticipates an increase in these actions, he wrote in response to questions from Type and The Intercept, because “the online nature of activism and speech generally makes it easier and more common to widely disseminate false claims and inflict great harm.”

“The claims against Greenpeace and others are … essentially saying, ‘Your activism is racketeering.’”

The cases against Greenpeace took the RICO strategy well beyond the arguments made by Chevron and Drummond. They argued that common advocacy techniques such as naming-and-shaming campaigns and fundraising amounted to RICO offenses if the campaigns included false allegations. Greenpeace’s campaign against Resolute included an inaccurate claim that Resolute had logged in protected forests, which Greenpeace later retracted, saying it had made a mistake. Resolute accused Greenpeace of intentionally fabricating the claim in order to extort the company, calling the organization a “global fraud” that existed to maximize donations rather than protect the environment.

“The claims against Donziger aren’t claims against environmentalism as it operates,” said Joshua Galperin, an environmental law professor at Pace Law School. “But the claims against Greenpeace and others are much more broad, essentially saying, ‘Your activism is racketeering.’”

Bowe disputed this characterization. “The case is not about activism, it is about lies,” he wrote. “Legitimate activism is truthful.”

Krystal Two Bulls, an organizer who participated in the Standing Rock protests against the Dakota Access pipeline, was added as a defendant in the racketeering suit brought by Energy Transfer in 2018, after a judge found that the initial complaint was too vague to support RICO claims. The company charged that Two Bulls, a media liaison for a group of protesters called Red Warrior Camp, had sought to “provide cover for their illegal activities” by issuing public calls to action on the group’s behalf. They accused Red Warrior Camp of being a “front for eco-terrorists” who engaged in violent attacks on construction sites. News reports state that while members of the camp occupied private land to block pipeline construction, police and security guards carried out much of the violence — using water hoses, rubber bullets, and tear gas against protesters.

Two Bulls, a U.S. Army veteran and a member of the Oglala Lakota and Northern Cheyenne, was shocked when she learned she had been charged with racketeering.

“I remember thinking, what am I supposed to do with this?” she said. “I have no lawyer. I have no money for a lawyer.”

“I started to censor myself.”

Two Bulls was represented pro bono by the nonprofit law firms Center for Constitutional Rights and EarthRights International. She considers herself lucky that colleagues in the environmental movement connected her with these lawyers but recalls a heavy weight on her shoulders while the charges were pending. She felt like her presence was a liability to her fellow activists.

“It made me second guess myself and the spaces I entered,” Two Bulls said. “I started to censor myself in the things I was saying.”

Laura Lee Prather, a partner at Haynes Boone who specializes in First Amendment law, said civil RICO claims often lead to extended litigation because they depend heavily on the facts of the case. Defamation charges can be thrown out if the defendant can affirmatively show their statements were true. By contrast, a civil RICO claim usually requires a more complex defense.

“Civil RICO is much more difficult to have a court feel comfortable dismissing at any early stage,” Prather said.

Federal judges in California and North Dakota dismissed the RICO claims in both cases almost a year and a half after they were filed. In the Resolute case, the judge ruled that the company failed to prove that Greenpeace’s fundraising claims had directly caused the alleged harm it suffered. He later ordered the company to pay more than $800,000 of Greenpeace’s legal costs.

Resolute noted that other charges it has brought against Greenpeace, alleging defamation and unfair competition, were allowed to proceed and are still before the courts. “The long-running dispute with activists has been about standing up for our communities to defend our sustainable practices against misrepresentation,” Resolute spokesperson Seth Kursman said in a statement.

In the case of Energy Transfer, the judge ruled that the company failed to prove that the various actors involved in the Standing Rock protests were a coordinated “RICO enterprise.”

“While there is a common purpose among defendants — they all oppose the Dakota Access Pipeline — there is no ongoing organization, no continuing unit, and no ascertainable structure apart from the alleged RICO violations,” U.S. District Judge Billy Roy Wilson wrote in February 2019. “That is far short of what is needed to establish a RICO enterprise.”

Energy Transfer did not respond to email or telephone inquiries. A week after its RICO charges were dismissed, the company filed charges in North Dakota state court, accusing Greenpeace, Two Bulls, and others of trespass, defamation, and civil conspiracy for their role in the Standing Rock protests. The litigation is ongoing.

Dakota Access pipeline protesters face off with various law enforcement agencies on Feb. 22, 2017, in North Dakota.

Photo: Michael Nigro/Pacific Press/LightRocket via Getty Images

Protecting the Protest

The RICO attacks on Greenpeace and its allies alarmed civil society organizations, which feared that the cases would deter advocacy groups from speaking out against big corporations. In 2018, a coalition of organizations founded Protect the Protest to combat lawsuits meant to silence free speech, which are known as strategic lawsuits against public participation, or SLAPPs. These lawsuits can include RICO claims but have also proliferated in other ways. Telltale signs of a SLAPP, according to the coalition, are claims that target activities protected by the First Amendment, seek to exploit a power imbalance, and threaten to bankrupt defendants.

“Civil society is not just going to lay down and take this,” said Marco Simons, the general counsel for EarthRights International and a member of the coalition.

Simons believes the coalition’s recent work calling attention to the Greenpeace lawsuits has, for the time being, discouraged companies from attempting more RICO suits that broadly target activism. But Protect the Protest is still seeking more permanent solutions.

The coalition aims to crack down on these suits by promoting anti-SLAPP laws, which provide fast-track procedures for dismissing SLAPPs and shifting their legal costs to the party that filed them. More than half of U.S. states have some version of an anti-SLAPP law.

Ken White, the former prosecutor, said that state anti-SLAPP laws have been highly effective, both in deterring abusive lawsuits and providing a defense mechanism for their targets. But RICO is a federal law.

In September, Rep. Jamie Raskin, D-Md., introduced the SLAPP Protection Act of 2022, a federal bill that, like the state laws, would provide an expedited process for getting SLAPPs thrown out. Raskin singled out the fossil fuel industry for abusing the “legal system by deploying costly, protracted, and meritless lawsuits to target activists.”

A law providing a uniform standard for dismissing such lawsuits across federal courts would make it “much harder to abuse the system,” White said.

As advocates search for solutions, Drummond is pressing ahead with its RICO case against Collingsworth. The company subpoenaed VICE Media last year for raw audio recordings from a podcast about the union leaders’ murders. On March 7, Proctor, the judge, ruled in Drummond’s favor, ordering VICE to turn over recordings of its interviews with Collingsworth, Blanco, and another witness.

Collingsworth said that he doesn’t fear losing in court, but the looming racketeering charges have taken a toll psychologically.

“There is a question mark over my name.”

“It has caused me emotional turmoil because some people view me differently,” he said. “There is a question mark over my name.”

The coming months are expected to bring new developments in his legal battle with the coal company. Attorneys will take depositions from witnesses in Colombia for Drummond’s RICO suit and a more recent suit brought by Collingsworth. Meanwhile, Colombian prosecutors have resumed work in their case against the current and former presidents of Drummond’s Colombian subsidiary, seeking testimony from a former paramilitary leader in October. The defendants have appealed the decision to charge them with conspiring in Orcasita’s and Locarno’s murders, and the appeal must be decided before the case can go to trial, according to Ivan Otero, Collingsworth’s co-counsel in Colombia.

More than 21 years after her husband’s murder, Elisa Orcasita is still skeptical of Colombian justice but is hoping for a clean trial.

“We pray to God that there’s no more buying of anything, no more influence of anything,” she said. “That’s what we hope for as victims.”

This story was produced with support from the Fund for Constitutional Government and the H.D. Lloyd Fund for Investigative Journalism.

The post The Scorched-Earth Legal Strategy Corporations Are Using to Silence Their Critics appeared first on The Intercept.

Melvyn Bragg and guests discuss the speeches that became a byword for fierce attacks on political opponents. It was in the 4th century BC, in Athens, that Demosthenes delivered these speeches against the tyrant Philip II of Macedon, father of Alexander the Great, when Philip appeared a growing threat to Athens and its allies and Demosthenes feared his fellow citizens were set on appeasement. In what became known as The Philippics, Demosthenes tried to persuade Athenians to act against Macedon before it was too late; eventually he succeeded in stirring them, even if the Macedonians later prevailed. For these speeches prompting resistance, Demosthenes became famous as one of the Athenian democracy’s greatest freedom fighters. Later, in Rome, Cicero's attacks on Mark Antony were styled on Demosthenes and these too became known as Philippics.

The image above is painted on the dome of the library of the National Assembly, Paris and is by Eugene Delacroix (1798-1863). It depicts Demosthenes haranguing the waves of the sea as a way of strengthening his voice for his speeches.

With

Paul Cartledge A. G. Leventis Senior Research Fellow at Clare College, University of Cambridge

Kathryn Tempest Reader in Latin Literature and Roman History at the University of Roehampton

And

Jon Hesk Reader in Greek and Classical Studies at the University of St Andrews

Producer: Simon Tillotson

Krystal and Saagar discuss Desantis vs Trump, Fed Rates, SEC catching Influencers, Marshall Law texts from Jan 6th, Elon banning Private Jet account, a bipartisan bill to ban Tik Tok, Revolutionary Labor moments of the year, an exclusive interview with Matt Taibbi on the Twitter Files and our upcoming live show in Austin Texas!

To become a Breaking Points Premium Member and watch/listen to the show uncut and 1 hour early visit: https://breakingpoints.supercast.com/

To listen to Breaking Points as a podcast, check them out on Apple and Spotify

Apple: https://podcasts.apple.com/us/podcast/breaking-points-with-krystal-and-saagar/id1570045623

Spotify: https://open.spotify.com/show/4Kbsy61zJSzPxNZZ3PKbXl

Merch: https://breaking-points.myshopify.com/

Learn more about your ad choices. Visit megaphone.fm/adchoices

Nearly five years after President Donald Trump broke with decades of U.S. policy and international consensus to recognize Jerusalem as Israel’s capital and move the U.S. Embassy there, the Biden administration is moving ahead with plans to build a permanent embassy compound in the city. Israel’s government has its headquarters in Jerusalem, but, because Palestinians also claim the city as their capital and because the city’s status remains disputed under international law, the U.S. Embassy, like those of most other countries, was previously based in Tel Aviv.

The plans for a new embassy, which the administration has quietly advanced in recent weeks, would consolidate Trump’s abrupt policy reversal and violate U.S. precedent both on the status of Jerusalem and on Israel’s ongoing illegal appropriation of Palestinian land. The new embassy would also make the U.S. government an active participant in that appropriation: The planned compound is to be built on land illegally expropriated from Palestinians, whose descendants, including several U.S. citizens, still have a claim to.

“The descendants of the landowners are all entitled to these properties under international law,” Suhad Bishara, legal director of the Israel-based human rights group Adalah, told The Intercept. “By moving forward with the plan to build the embassy at the Allenby site, the U.S. are taking an active part in the illegal confiscation of these properties, including infringing on their own citizens’ rights.”

“Ordinary Americans should have a chance to decide: Do we want our government to take property stolen from U.S. citizens for a U.S. Embassy?”

According to plans submitted by the U.S. State Department to Israeli authorities, the diplomatic compound would be built on a grassy plot known as the “Allenby Barracks,” after a former British military camp that the British leased from Palestinian families. The land is now registered to the state of Israel and leased to the U.S. after it was confiscated from Palestinian refugees under Israel’s 1950 Absentees’ Property Law — widely condemned legislation that has enabled Israel to claim ownership over the land of countless Palestinians it displaced.

“This land was illegally confiscated,” Bishara said, adding that absentees’ property laws were “racially designed to confiscate Palestinian property for the benefit of Israel and its Judaization processes in the area.”

In a statement sent to The Intercept after this article was published, Rachel Rubin, a spokesperson for the State Department, said, “The United States has not yet made a decision on which site to pursue. A number of factors, including the history of the sites, will be part of our site selection process.”

Rubin confirmed that the Biden administration would uphold Trump’s embassy move: “The U.S. position is that our embassy will remain in Jerusalem, which we recognize as Israel’s capital.”

Plans submitted by the U.S. State Department for a new U.S. embassy compound in Jerusalem.

Source: Israeli planning authorities

Descendants of those landowners have been raising their well-documented claim to that land at least since U.S. plans for the plot were first floated, and then abandoned, in the 1990s. The U.S. government has been aware of the claims to the land at least since then. Earlier this year, researchers at Adalah uncovered additional archival documentation, including deeds that remove any doubt about the land’s rightful owners.

One of the Americans with claims to the land where the U.S. Embassy is to be built is Rashid Khalidi, a respected historian and Columbia University professor whose family is one of several that has called on the U.S. government to cancel the plans. The families have also asked for a meeting with U.S. Secretary of State Antony Blinken and U.S. Ambassador to Israel Tom Nides, but have received no response. (Rubin, the State Department spokesperson, said she would not comment on private correspondence, though the letter on behalf of the heirs to the land is public.)

“This is a country which supposedly considers private property sacred,” Khalidi told The Intercept. “Why is the private property of Palestinian Americans — Americans who happen to be Palestinian — something that the U.S. government feels it can allow a foreign government to take and then lease that land from that foreign government?”

“They have done this quietly. They’ve not trumpeted it,” he added. “But ordinary Americans should have a chance to decide: Do we want our government to take property stolen from U.S. citizens for a U.S. Embassy?”

Israel’s Prime Minister Benjamin Netanyahu, left, his wife Sara Netanyahu, Ivanka Trump, and Jared Kushner take a selfie at the opening of the U.S. Embassy in Jerusalem on May 14, 2018.

Photo: Israel Press Office /Getty Images

An American Plan

So far, U.S. officials have not privately or publicly acknowledged claims to the Allenby Barracks land by the original owners’ descendants, including the ones who are U.S. citizens.

In public statements, U.S. State Department officials have indicated that they are still deliberating over plans for the new diplomatic compound and doing “due diligence” on prospective sites. In addition to the Allenby Barracks plot, they are also considering a second site in the affluent Jerusalem neighborhood of Arnona, near the site of the current temporary U.S. Embassy.

According to official Israeli transcripts, however, U.S. officials have told Israeli planning authorities that they intend to develop both plots. “One complex will be for the Embassy’s office building and the other complex will be used for other uses, and will be developed after the Embassy is built,” State Department representatives told Israeli officials, according to one Hebrew-language transcript, suggesting that the second site might be used to house diplomats and embassy staff.

The U.S. government has invested significantly in the plans for the disputed site. Last year, during a Zoom presentation before Israeli officials, including the mayor of Jerusalem, four State Department representatives pitched their plan for the new embassy — making no reference to the land’s disputed status but describing the boost to commerce that the compound would bring to the area. The proposal included a slideshow with 3-D renderings of the multi-building plan, detailed enough to include references to traffic and parking impact as well as plans for “tree preservation.”

Official planning documents for the compound were filed with Israeli authorities by James Kania, a U.S. foreign service officer who listed on his LinkedIn page overseeing “real estate and construction projects including a $17 million Ambassador’s Residence and a $1 million retrofit of a residence for the U.S. Marine Detachment in Jerusalem” as well as having “served as the main logistics manager for the physical transition of the U.S. Embassy from Tel Aviv to Jerusalem.” (Kania declined to comment, referring questions to the embassy.)

The architects for both the Allenby and Arnona projects are Chicago-based Krueck Sexton Partners, in collaboration with an Israel-based firm. Krueck Sexton Partners did not respond to a request for comment.

Despite U.S. officials’ vague statements on the matter, the plans and transcripts recently made public by Israeli authorities leave no doubt that the State Department is not only moving forward with the new embassy, but that it is also actively lobbying for Israel’s sign-off despite repeated objections by the land’s rightful owners.

“They’re trying to do this on the down-low,” said Khalidi. “They pretend that they’re not involved. In fact, the planning documents are prepared by the U.S. government. One of them has the U.S. Embassy in Israel logo on it. It’s a fiction. This is a U.S. government effort — together with the Israeli planning authorities, of course.”

A view of the border fence between the Israel-annexed Golan Heights and Syria, on Sept. 19, 2022 .

Photo: Jalaa Marey/AFP via Getty Images

Stolen Land

Some 750,000 Palestinians were expelled or forced to flee, becoming refugees, when Israel was established in 1948. For decades, Israel has refused their return and confiscated their lands, even as it continues to illegally expand and build settlements in Palestinian territories it has occupied more recently.

The international community, including the U.S., has long condemned Israel’s expropriation of Palestinian land as a violation of refugee and property rights. The U.S. voted in favor of a United Nations resolution asserting Palestinian refugees’ right of return to their homes and compensation for those choosing not to return.

“The question is, is this going to be made permanent by this administration’s policy?”

The U.S. has effectively done nothing, however, to stop the expropriation of Palestinian land — including the ongoing construction of settlements in the Israeli-occupied West Bank and East Jerusalem. To build a U.S. Embassy on stolen Palestinian land, critics say, would deal yet another blow to already lagging U.S. legitimacy in the region.

Until Trump, the U.S. has also maintained, along with much of the rest of the world, that the status of Jerusalem should be decided in line with U.N. resolutions and through negotiations — rejecting Israel’s unilateral declarations of sovereignty over the city. Trump shocked the world when he broke with decades of U.S. precedent by recognizing Israel’s annexation of Jerusalem and of the Golan Heights, a Syrian region abutting Israel’s border that has been under Israeli occupation since 1967. The Biden administration has remained largely quiet on Trump’s moves.

“They have not said, ‘The United States continues to fully recognize the annexation of the Golan Heights, the United States continues to fully recognize the annexation of Jerusalem,’ as announced by the Trump administration, but in practice, they haven’t dissented from those policies,” said Khalidi. “The question is, is this going to be made permanent by this administration’s policy?”

Update: December 15, 2022, 5:20 p.m. ET
This story has been updated to include a statement from the State Department received after publication.

The post Biden Moves Ahead on Trump Plan to Build Israel Embassy on Stolen Palestinian Land appeared first on The Intercept.

We review the year in Podcasts, according to the New York Times. Then, Sinema flips, the Young Republicans present the Allen Dulles award, and the CIA has a mental wellness problem. Finally, some movie mindset recommendations for your holiday season.

Hell on Earth launch show/party @ Littlefield 1/20/23, tickets here: https://littlefieldnyc.com/event/?wfea_eb_id=479703214227

Matt w/ Talking Simpsons podcast, Live at SF Sketchfest, 1/25/23, tickets here: https://www.eventbrite.com/e/talking-simpsons-tickets-210772966617

Hell on Earth Bibliography/Reading List Here: https://www.chapotraphouse.com/hell

Hosted on Acast. See acast.com/privacy for more information.

So, there's this website, Have I Been Pwned, where you can check if your email address has appeared in a data breach.

There's also a Pwned Passwords section for passwords ... but, typing your password on a random website probably isn't such a great idea, right?

Of course, you could read about how HIBP protects the privacy of searched passwords, and understand how k-Anonymity works, and check that the website uses the k-Anonymity API, but that's waaay too much work.

Of course, you could use the API yourself, but where's the fun in that?

Instead, we'll do it the hard way – we'll check passwords offline.

And we're not stopping until it's fast.

• The Pwned Passwords list
• A minimal plausible solution
• Problem: it's slow
  • Skipping
• Problem: it needs tuning, it's still slow
  • Binary skipping
  • Better timing
• Failing to get to under 1 millisecond
  • Profile before optimizing
  • Position heuristic
  • Index file
  • Binary file
• Getting to under 1 millisecond
  • Generating the index
  • Using the index
  • I heard you like indexes (the end)
• Bonus: better data structures

The Pwned Passwords list #

OK, first we need the password list.

Go to Pwned Passwords, scroll to Downloading the Pwned Passwords list, and download the SHA-1 ordered-by-hash file – be nice and use the torrent, if you can.

Note

You can also use the downloader to get an updated version of the file, but that didn't exist when I started writing this.

The archive extracts to a 37 GB text file:

$ pushd ~/Downloads
$ stat -f %z pwned-passwords-sha1-ordered-by-hash-v8.7z
16257755606
$ 7zz e pwned-passwords-sha1-ordered-by-hash-v8.7z
$ stat -f %z pwned-passwords-sha1-ordered-by-hash-v8.txt
37342268646
$ popd

... that looks like this:

$ head -n5 ~/Downloads/pwned-passwords-sha1-ordered-by-hash-v8.txt
000000005AD76BD555C1D6D771DE417A4B87E4B4:10
00000000A8DAE4228F821FB418F59826079BF368:4
00000000DD7F2A1C68A35673713783CA390C9E93:873
00000001E225B908BAC31C56DB04D892E47536E0:6
00000006BAB7FC3113AA73DE3589630FC08218E7:3

Each line has the format:

<SHA-1 of the password>:<times it appeared in various breaches>

Note

For more details: why hash the passwords, and why include a count.

To make commands shorter, we link the file in the current directory:

$ ln -s ~/Downloads/pwned-passwords-sha1-ordered-by-hash-v8.txt pwned.txt

A minimal plausible solution #

We'll take an iterative, problem-solution approach to this. But since right now we don't have any solution, we start with the simplest thing that could possibly work.

First, we get the imports out of the way:

1
2
3
4
5

import os
import sys
import time
import getpass
import hashlib

Then, we open the file:

7
8

path = sys.argv[1]
file = open(path, 'rb')

Opening the file in binary mode makes searching a bit faster, as it skips needless decoding. By opening it early, we get free error checking – no point in reading the password if the file isn't there.

Next, the password:

10
11
12
13
14
15
16
17

try:
    password = sys.argv[2]
except IndexError:
    password = getpass.getpass()
hexdigest = hashlib.sha1(password.encode()).hexdigest()
del password

print("looking for", hexdigest)

We either take the password as the second argument to the script, or read it with getpass(), so real passwords don't remain in the shell history. After computing its hash, we delete it; we don't go as far as to zero‍¹ it, but at least we avoid accidental printing.

Let's see if it works:

$ python pwned.py pwned.txt password
looking for 5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
$ python pwned.py pwned.txt
Password:
looking for 5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8

sha1sum seems to agree:

$ echo -n password | sha1sum
5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8  -

To find the hash, we go through the file line by line – remember, simplest thing that could possibly work.

 8
 9
10
11
12
13
14

def find_line(lines, prefix):
    for line in lines:
        if line.startswith(prefix):
            return line
        if line > prefix:
            break
    return None

If a line was found, we print the count:

29
30
31
32
33
34
35

line = find_line(file, hexdigest.upper().encode())

if line:
    times = int(line.decode().rstrip().partition(':')[2])
    print(f"pwned! seen {times:,} times before")
else:
    print("not found")

Finally, let's add some timing code:

29
30
31

start = time.monotonic()
line = find_line(file, hexdigest.upper().encode())
end = time.monotonic()

39

print(f"in {end-start:.6f} seconds")

And, it works:

$ python pwned.py pwned.txt blocking
looking for 000085013a02852372159cb94101b99ccaec59e1
pwned! seen 587 times before
in 0.002070 seconds

The code so far.

Problem: it's slow #

You may have noticed I switched from password to blocking. That's because I deliberately chose a password whose hash is at the beginning of the file.

On my 2013 laptop, password actually takes 86 seconds!

To put a lower bound on the time it takes to go through the file:

$ time python -c 'for line in open("pwned.txt", "rb"): pass'

real	1m28.325s
user	1m10.049s
sys	0m15.577s
$ time wc -l pwned.txt
 847223402 pwned.txt

real	0m51.729s
user	0m27.908s
sys	0m12.119s

There must be a better way.

Skipping #

There's a hint in the file name: the hashes are ordered.

That means we don't have to check all the lines. We can skip ahead repeatedly until we're past the hash, go back one step, and only check each line from there.

Lines are different lengths, so we can't skip exactly X lines without reading them. But we don't need to, any line that's a reasonable amount ahead will do.

17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

def skip_to_before_line(file, prefix, offset):
    old_position = file.tell()

    while True:
        file.seek(offset, os.SEEK_CUR)

        file.readline()
        line = file.readline()
        # print("jumped to", (line or b'<eof>').decode().rstrip())

        if not line or line >= prefix:
            file.seek(old_position)
            break

        old_position = file.tell()

So we just seek() ahead a set number of bytes. Since that might not leave us at the start of a line, we discard the incomplete line, and use the next one.

Finally, we wrap the original find_line() to do the skipping:

 8
 9
10
11
12
13
14

def find_line(file, prefix):
    skip_to_before_line(file, prefix, 16 * 2**20)
    return find_line_linear(file, prefix)


def find_line_linear(lines, prefix):
    for line in lines:

It works:

$ python pwned.py pwned.txt password
looking for 5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
pwned! seen 9,545,824 times before
in 0.027203 seconds

To find the magic 16M offset, I had to try a bunch values:

offset (MiB)   time (s)
           1      0.05
           4      0.035
           8      0.030
          16      0.027  <-- sweet spot
          32      0.14

The code so far.

Problem: it needs tuning, it's still slow #

While three orders of magnitude faster, we still have a bunch of issues:

1. The ideal offset depends on the computer you're running this on.
2. The run time still increases linearly with file size – we haven't really solved the problem, as much as made it smaller by a (large, but) constant factor.
3. The run time still increases linearly with where the hash is in the file.
4. It's still kinda slow. ¯\_(ツ)_/¯

There must be a better way.

Binary skipping #

To make the "linear" part painfully obvious, uncomment the jumped to line.

$ python pwned.py pwned.txt password | grep -o 'jumped to .' | uniq -c
 139 jumped to 0
 139 jumped to 1
 139 jumped to 2
 139 jumped to 3
 139 jumped to 4
 103 jumped to 5

Surely, after we've jumped to 0 once, we don't need to do it 138 more times, right?

We could jump directly to a line in the middle of the file; if there at all, the hash will be in either of the halves. We could then jump to the middle of that half, and to the middle of that half, and so on, until we either find the hash or there's nowhere left to jump.

Note

If that sounds a lot like binary search, that's because it is – it's just not wearing its regular array clothing.

And most of the work is already done: we can jump to a line at most X bytes from where the hash should be, we only need to do it repeatedly, in smaller and smaller fractions of the file size:

25
26
27
28
29
30
31
32

def skip_to_before_line(file, prefix, offset):
    while offset > 2**8:
        offset //= 2
        skip_to_before_line_linear(file, prefix, offset)


def skip_to_before_line_linear(file, prefix, offset):
    old_position = file.tell()

The only thing left is to get the file size:

 8
 9
10
11
12
13

def find_line(file, prefix):
    file.seek(0, os.SEEK_END)
    size = file.tell()
    file.seek(0)
    skip_to_before_line(file, prefix, size)
    return find_line_linear(file, prefix)

It works:

$ python pwned.py pwned.txt password
looking for 5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
pwned! seen 9,545,824 times before
in 0.009559 seconds
$ python pwned.py pwned.txt password
looking for 5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
pwned! seen 9,545,824 times before
in 0.000268 seconds

The huge time difference is due to operating system and/or disk caches – on the second run, the same parts of the file are likely already in memory.

Anyway, look again at the jumped to output: instead of jumping blindly through the whole file, now we're jumping around the hash, getting closer and closer to it.

$ python pwned.py pwned.txt password | grep -o 'jumped to .' | uniq -c
   1 jumped to 7
   1 jumped to 3
   1 jumped to 7
   1 jumped to 5
   1 jumped to 4
  39 jumped to 5

$ python pwned.py pwned.txt password | grep -o 'jumped to ..' | uniq -c
   1 jumped to 7F
   1 jumped to 3F
   1 jumped to 7F
   1 jumped to 5F
   1 jumped to 4F
   1 jumped to 5F
   1 jumped to 57
   1 jumped to 5F
   1 jumped to 5B
   1 jumped to 59
   1 jumped to 5B
   1 jumped to 5A
  32 jumped to 5B

Notice we land on the same 7F... prefix twice; this makes sense – we skip ahead by half the file size, then back, then ahead two times by a quarter. Caching allows us to not care about that.

The code so far.

Better timing #

Given the way caching muddies the waters, how fast is it really?

This function averages a hundred runs, each with a different password:

function average-many {
    for _ in {1..100}; do
        python $@ $( python -c 'import time; print(time.time())' )
    done \
    | grep seconds \
    | cut -d' ' -f2 \
    | paste -sd+ - \
    | sed 's#^#scale=6;(#' | sed 's#$#)/100#' \
    | bc
}

$ for _ in {1..20}; do average-many pwned.py pwned.txt; done
.004802
.003904
.004088
.003486
.003451
.003476
.003414
.003442
.003169
.003297
.002931
.003077
.003092
.003011
.002980
.003147
.003112
.002942
.002984
.002934

Again, this is due to caching: the more we run the script, the more likely it is that the pages at {half, quarters, eights, sixteenths, ...} of the file size are already in memory – and the path to any line starts with a subset of those.

And, we're done.

I wave my hands, get a 2020 laptop, and a miracle happens. It's far enough into the totally unpredictable future, now, that you can search any password in under 1 millisecond. You can do anything you want.

So, there we go. Wasn't that an interesting story? That's the end of the article.

Don't look at the scroll bar. Don't worry about it.

If you came here to check your password, you can go.

Subscribe on the way out if you'd like, but take care :)

Go solve Advent of Code or something.

I'm just chilling out.

See ya.

Want to know when new articles come out? Subscribe here to get new stuff straight to your inbox!

Failing to get to under 1 millisecond #

I swear this was supposed to be the end. This really was supposed to be a short one.

Here's a quote from a friend of mine, that chronologically should be way later into the article, but makes a great summary for what follows:

And now that you have arrived at this point, spend a moment to ponder the arbitrary nature of 1 millisecond given its dependency on the current year and the choice of your particular hardware.

After that moment, continue celebrating.

Nah, fuck it, it has to take less than 1 millisecond on the old laptop.

... so yeah, here's a bunch of stuff that didn't work.

Profile before optimizing #

With the obvious improvements out of the way, it's probably a good time to stop and find out where time is being spent.

$ python -m cProfile -s cumulative pwned.py pwned.txt "$( date )"
looking for 3960626a8c59fe927d3cf2e991d67f4c505ae198
not found
in 0.004902 seconds
         1631 function calls (1614 primitive calls) in 0.010 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      ...
        1    0.000    0.000    0.005    0.005 02-binary-search.py:8(find_line)
        1    0.000    0.000    0.005    0.005 02-binary-search.py:22(skip_to_before_line)
       28    0.000    0.000    0.005    0.000 02-binary-search.py:28(skip_to_before_line_linear)
       86    0.004    0.000    0.004    0.000 {method 'readline' of '_io.BufferedReader' objects}
      ...
       71    0.000    0.000    0.000    0.000 {method 'seek' of '_io.BufferedReader' objects}
      ...
       44    0.000    0.000    0.000    0.000 {method 'tell' of '_io.BufferedReader' objects}
      ...

From the output above, we learn that:

• Most of the time is spent in readline() calls.
• Both seek() and tell() calls are basically free.

readline() is implemented in C, so there's not much we can change there.

What we can change, however, is how often we call it.

Another thing of interest is how much individual readline() calls take.

In skip_to_before_line_linear():

34
35
36
37
38
39
40

        start = time.monotonic()
        file.readline()
        line = file.readline()
        end = time.monotonic()
        print("jumped to", (line[:5] or b'<eof>').decode().rstrip(),
              f"in {(end-start)*1000000:4.0f} us",
              f"at offset {file.tell():16,}")

$ python pwned.py pwned.txt asdf
looking for 3da541559918a808c2402bba5012f6c60b27661c
jumped to 7FF9E in   10 us at offset   18,671,134,394  <-- 1/2 file size
jumped to 3FF89 in    4 us at offset    9,335,567,234  <-- 1/4 file size
jumped to 1FFBA in    3 us at offset    4,667,783,663  <-- 1/8 file size
jumped to 3FF89 in    3 us at offset    9,335,567,322  <-- 1/4 file size
jumped to 2FFA4 in    5 us at offset    7,001,675,508
jumped to 3FF89 in    4 us at offset    9,335,567,366  <-- 1/4 file size
jumped to 37F98 in    4 us at offset    8,168,621,453
jumped to 3FF89 in    3 us at offset    9,335,567,410  <-- 1/4 file size
jumped to 3BF94 in    3 us at offset    8,752,094,477
jumped to 3FF89 in    2 us at offset    9,335,567,498  <-- 1/4 file size
jumped to 3DF8E in    3 us at offset    9,043,831,007
jumped to 3CF90 in    3 us at offset    8,897,962,782
jumped to 3DF8E in    2 us at offset    9,043,831,095
jumped to 3D790 in    3 us at offset    8,970,896,964
jumped to 3DF8E in    2 us at offset    9,043,831,139
jumped to 3DB90 in  253 us at offset    9,007,364,072
jumped to 3D990 in  206 us at offset    8,989,130,552
jumped to 3DB90 in    6 us at offset    9,007,364,160
jumped to 3DA8F in  270 us at offset    8,998,247,402  <-- page 2,196,837
jumped to 3DA0F in  189 us at offset    8,993,689,007
jumped to 3DA8F in    5 us at offset    8,998,247,446  <-- page 2,196,837
jumped to 3DA4F in  212 us at offset    8,995,968,274
jumped to 3DA8F in    5 us at offset    8,998,247,534  <-- page 2,196,837
jumped to 3DA6F in  266 us at offset    8,997,107,921
jumped to 3DA5F in  203 us at offset    8,996,538,139
jumped to 3DA57 in  195 us at offset    8,996,253,241
jumped to 3DA53 in  197 us at offset    8,996,110,772
jumped to 3DA57 in    6 us at offset    8,996,253,285
jumped to 3DA55 in  193 us at offset    8,996,182,045
jumped to 3DA54 in  178 us at offset    8,996,146,471
jumped to 3DA54 in  189 us at offset    8,996,128,666
jumped to 3DA54 in  191 us at offset    8,996,119,760  <-- page 2,196,318
jumped to 3DA54 in   32 us at offset    8,996,128,710
jumped to 3DA54 in    5 us at offset    8,996,124,259
jumped to 3DA54 in   10 us at offset    8,996,122,057  <-- page 2,196,318
jumped to 3DA54 in    4 us at offset    8,996,120,955  <-- page 2,196,318
jumped to 3DA54 in    4 us at offset    8,996,120,382  <-- page 2,196,318
jumped to 3DA54 in    9 us at offset    8,996,120,112  <-- page 2,196,318
jumped to 3DA54 in    1 us at offset    8,996,120,470  <-- page 2,196,318
jumped to 3DA54 in    1 us at offset    8,996,120,338  <-- page 2,196,318
pwned! seen 324,774 times before
in 0.003654 seconds

Half the reads are pretty fast:

• In the beginning, because searches start with the same few pages.
• At the end, because searches end on the same page.
• All reads of any page, after the first.

So, it's those reads in the middle that we need to get rid of.

Position heuristic #

In theory, the output of a good hash function should be uniformly distributed.

This means that with a bit of math, we can estimate where a hash would be – a hash that's ~1/5 in the range of all possible hashes should be at ~1/5 of the file.

Here's a tiny example:

>>> digest = '5b'  # 1-byte hash (2 hex digits)
>>> size = 1000    # 1000-byte long file
>>> int_digest = int(digest, 16)  # == 91
>>> int_end = 16 ** len(digest)   # == 0xff + 1 == 256
>>> int(size * int_digest / int_end)
355

We can do this once, and then binary search a safety interval around that position. Alas, this only gets rid of the fast jumps at the beginning of the binary search, and for some reason, it ends up being slightly slower than binary search alone. (code)

We can also narrow down around the estimated position iteratively, making the interval smaller by a constant factor each time. This seems to work: a factor of 1000 yields 1.7 ms, and a factor of 8000 yields 1.2 ms, both in 2 steps. (code)

However, it has deeper issues:

• Having arbitrary start/end offsets complicates the code quite a bit.
• I don't know how to reliably determine the factor.²
• I don't know how to prove it's correct,³ especially for smaller intervals, where the hashes are less uniform. To be honest, I don't think it can be 100% correct, and I don't know how to estimate how correct it is.

Anyway, if the implementation is hard to explain, it's a bad idea.

Index file #

An arbitrary self-imposed restriction I had was that any solution should mostly use the original passwords list, with little to no preparation.

By relaxing this a bit, and going through the file once, we can build an index like:

<SHA-1 of the password>:<offset in file>

... that we can then search with skip_to_before_line().

Of course, we won't include all the hashes – by including lines a few kilobytes apart from each other, we can seek directly to within a few kilobytes in the big file.

The only thing left to figure out is how much "a few kilobytes" is.

After my endless harping about caching and pages, the answer should be obvious: one page size (4K). And this actually gets us 0.8 ms! But back when I wrote the code, that hadn't really sunk in, so after getting 1.2 ms with a 32K distance, I moved on.

Code: pwned.py, generate-index.py.

Binary file #

Already on the additional file slippery slope, I converted the list to binary, mostly to make it smaller – smaller file, fewer reads.

I packed each line into 24 bytes:

| binary hash (20 bytes) | count (4 bytes) |

This halved the file, but only lowered the runtime to a modest 2.6 ms.

More importantly, it made the code much, much simpler: because items are fixed size, you can know where the Nth item is, so I was able to use bisect for the binary search.

Code: pwned.py, convert-to-binary.py.

Getting to under 1 millisecond #

OK, what now? This is what we have so far:

• The position heuristic kinda (maybe?) works, but is hard to reason about.
• The index file gets us there, but barely, and the index is pretty big.
• The binary file isn't much faster, and it creates a huge file. But, less code!

I don't know what to do with the first one, but I think we can combine the last two.

Generating the index #

Let's start with the script I made for the text index:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import os, sys

file = sys.stdin.buffer
outf = sys.stdout.buffer

while True:
    pos = file.tell()
    line = file.readline()
    if not line:
        break

    outf.write(line.partition(b':')[0] + b':' + str(pos).encode() + b'\n')

    file.seek(2**12, os.SEEK_CUR)
    file.readline()

The output looks like this:

$ python generate-index.py < pwned.txt 2>/dev/null | head -n5
000000005AD76BD555C1D6D771DE417A4B87E4B4:0
00000099A4D3034E14DF60EF50799F695C27C0EC:4157
00000172E8E1D1BD54AC23B3F9AB4383F291CA17:8312
000002C8F808A7DB504BBC3C711BE8A8D508C0F9:12453
0000047139578F13D70DD96BADD425C372DB64A9:16637

We need to pack that into bytes.

A hash takes 20 bytes. But, we only need slightly more than 3 bytes (6 hex digits) to distinguish between the index lines:

$ python generate-index.py < pwned.txt 2>/dev/null | cut -c-6 | uniq -c | head
   2 000000
   1 000001
   1 000002
   1 000004
   1 000005
   1 000007
   1 000008
   1 00000A
   1 00000B
   1 00000C

To represent all the offsets in the file, we need log2(35G) / 8 = 4.39... bytes, which results in a total of 9 bytes (maybe even 8, if we mess with individual bits).

Let's make it future-proof: 6 bytes for the hash buys at least 55 trillion lines (2.4 petabyte files), and 6 bytes for the offset buys 0.28 petabyte files.

12
13
14

    digest, _, _ = line.partition(b':')
    outf.write(bytes.fromhex(digest.decode())[:6])
    outf.write(pos.to_bytes(6, 'big'))

If you look at the text index, you'll notice the offsets are 4K + ~50 bytes apart; this results in sometimes having to read 2 pages, because not all pages have an index entry. Let's fix that by reading the first whole line after a 4K boundary instead:

16
17

    file.seek((pos // 4096 + 1) * 4096)
    file.readline()

OK, we're done:

$ time python generate-index.py < pwned.txt > index.bin

real	1m2.729s
user	0m34.292s
sys	0m21.392s

Using the index #

We start with a skeleton that's functionally identical to the naive script. The only difference is that I've added stubs for passing and using the index:

 9
10
11

def find_line(file, prefix, index):
    skip_to_before_line_index(file, prefix, index)
    return find_line_linear(file, prefix)

23
24

def skip_to_before_line_index(file, prefix, index):
    ...

30
31

index_path = sys.argv[2]
index = ...

43

line = find_line(file, hexdigest.upper().encode(), index)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

import os
import sys
import time
import bisect
import getpass
import hashlib


def find_line(file, prefix, index):
    skip_to_before_line_index(file, prefix, index)
    return find_line_linear(file, prefix)


def find_line_linear(lines, prefix):
    for line in lines:
        if line.startswith(prefix):
            return line
        if line > prefix:
            break
    return None


def skip_to_before_line_index(file, prefix, index):
    ...


path = sys.argv[1]
file = open(path, 'rb')

index_path = sys.argv[2]
index = ...

try:
    password = sys.argv[3]
except IndexError:
    password = getpass.getpass()
hexdigest = hashlib.sha1(password.encode()).hexdigest()
del password

print("looking for", hexdigest)

start = time.monotonic()
line = find_line(file, hexdigest.upper().encode(), index)
end = time.monotonic()

if line:
    times = int(line.decode().rstrip().partition(':')[2])
    print(f"pwned! seen {times:,} times before")
else:
    print("not found")

print(f"in {end-start:.6f} seconds")

As mentioned before, we'll use the standard library bisect module to search the index.

We could read the entire index in memory, as a list of 12-byte bytes. But that would still be slow, even if outside the current timing code, and memory usage would increase with the size of the file.

Fortunately, bisect doesn't only work with lists, it works with any sequence – that is, any object that can pretend to be a list. So we'll build our own, by implementing the two required⁴ special methods.

27
28
29
30
31
32

class BytesArray:

    item_size = 12

    def __init__(self, file):
        self.file = file

We can go ahead and plug it in:

38
39

index_path = sys.argv[2]
index = BytesArray(open(index_path, 'rb'))

The first special method is __getitem__(), for a[i]:

34
35
36
37
38
39

    def __getitem__(self, index):
        self.file.seek(index * self.item_size)
        buffer = self.file.read(self.item_size)
        if len(buffer) != self.item_size:
            raise IndexError(index)  # out of bounds
        return buffer

The second special method is __len__(), for len(a):

41
42
43
44

    def __len__(self):
        self.file.seek(0, os.SEEK_END)
        size = self.file.tell()
        return size // self.item_size

Using the index becomes straightforward:

23
24
25
26
27
28
29
30
31
32
33
34

def skip_to_before_line_index(file, prefix, index):
    item_prefix = bytes.fromhex(prefix.decode())[:6]
    item = find_lt(index, item_prefix)
    offset = int.from_bytes(item[6:], 'big') if item else 0
    file.seek(offset)


def find_lt(a, x):
    i = bisect.bisect_left(a, x)
    if i:
        return a[i-1]
    return None

We get the first 6 bytes of the hash, find the rightmost value less than that, extract the offset from it, and seek to there. find_lt() comes from bisect's recipes for searching sorted lists.

And we're done:

$ average-many pwned.py pwned.txt index.bin
.002546

Huh? ... that's unexpected...

Oh.

I said we won't read the index in memory. But we can force it into the cache by reading it a bunch of times:

$ for _ in {1..10}; do cat index.bin > /dev/null; done

Finally:

$ average-many pwned.py pwned.txt index.bin
.000421

Code: pwned.py, generate-index.py.

I heard you like indexes (the end) #

Hmmm... isn't that cold start bugging you? If we make an index for the big index, we get 1.2 ms from a cold start. Maybe another smaller index can take us to below 1 ms?

...

Just kidding, this is it, this really is the end.

And now, let's take that moment to ponder:

For twice the code, it's 5 orders of magnitude faster! I'm deliberately not counting bisect or the OS cache here, beacuse that's the point, they're basically free.

Turns out, you can get pretty far with just a few tricks.

That's it for now.

Learned something new today? Share this with others, it really helps!

Bonus: better data structures #

As always, a specialized data structure can solve a problem better.

In Sketchy Pwned Passwords, Scott Helme manages to "pack" the entire passwords list into a 1.5G in-memory count-min sketch, which can then be queried in 1 microsecond. And, if you don't care about the counts, a plain Bloom filter can even take you to 0.1 µs! (There is a trade-off, and it's that it takes 11 hours to create either.)

Want to know when new articles come out? Subscribe here to get new stuff straight to your inbox!

1. It's kinda difficult to do in Python anyway. ^[return]

2. Something like (size / 4096) ** (1 / int(log(size, 4096))), maybe? ^[return]

3. I mean, I did cross-check it with other solutions for a few thousand values, and it seems correct, but that's not proof. ^[return]

4. We're only implementing the parts of the protocol that bisect uses.

   For the full protocol, __getitem__() would need to implement negative indexes and slicing. For more, free sequence methods, inherit collections.abc.Sequence.

   Interestingly, the class will work in a for loop without an __iter__() method. That's because there are actually two iteration protocols: an older one, using __getitem__(), and a newer one, added in Python 2.1, using __iter__() and __next__(). For details on the logic, see Unravelling for statements. ^[return]

Pasta shelves are empty at a Save Mart supermarket during the pandemic in Porterville, Calif., in 2020.

Photo: Jeremy Hogan/Getty Images

I realize this question seems extremely boring: Has the recent bout of high inflation in the U.S. been caused by insufficient supply in various areas of the economy, or too much overall demand?

But please bear with me. Because the answer illuminates and affects every single aspect of the life you’re living, right now. Would you like your town’s jagoff employers to be so desperate that they apply for you to work for them, instead of the other way around? Would you like to defuse the underlying rage in American life that seems to lead to a mass shooting every 37 seconds? Would you like $50 trillion? (Not made up, see below.) Then keep reading.

A new paper by Ira Regmi and Joseph Stiglitz makes the case that the answer is the former, i.e., insufficient supply. As they put it, “Today’s inflation comes mostly from sectoral supply side disruptions, largely the result of the COVID-19 pandemic … and disruptions to energy and food markets originating from Russia’s invasion of Ukraine. … [It] is not the result of significant excesses of aggregate demand such as might have arisen from excessive U.S. pandemic spending.” This means large, fast increases in interest rates by the Federal Reserve “will not substantially lower inflation unless they induce a major contraction in the economy, which is a cure worse than the disease.”

Regmi is program manager for the macroeconomic analysis program at the Roosevelt Institute, a progressive think tank. Stiglitz is as fancy as economists get: He won the quasi-Nobel Prize in Economics in 2001; he’s now a professor at Columbia University; and he once was chief economist for the World Bank.

Ninety-five percent of Americans are unaware of the stakes of this issue. If Regmi and Stiglitz are correct, it suggests that we can use the tools of the government to make life better for almost everyone. If they’re wrong, perhaps that’s impossible — because the Immutable Laws of Economics simply don’t allow it — and if we try to make our lives better, we will be punished for our hubris.

The paper’s math is as basic as it gets and it’s lucidly written, so it should be comprehensible for any audience, even reporters. However, Regmi and Stiglitz are far too sober and responsible to include the powerful political relevance of their work. I’m not, though, so let’s start there.

Faithful consumers of America’s elite media — TV news, the New York Times, the Washington Post — generally get the impression that everyone in the U.S. is on the same team. While we may disagree on how to get there, we all share the same economic goals: a fast-growing economy with low unemployment and a thriving middle class, one whose members keep doing better than their parents.

This is absolutely false. The people at America’s commanding heights do not want this at all. And if you look at it from their perspective, that’s easy to understand. Low unemployment means an unruly workforce with the leverage and confidence to unionize. Rising wages for employees means less money for employers. The higher inflation that tends to accompany a high-demand economy of mass affluence is effectively a massive transfer of wealth from creditors to debtors. As an Ohio business owner once told the New York Times, “I sometimes wish there was actually a higher unemployment rate.”

So the uncomfortable truth is that the top 1 percent generally prefer a slower-growing economy with higher unemployment to its opposite. This is the case even though they might be richer in absolute terms in an economy that worked for everyone. But instead they prefer to have less money with more relative power. That preference may seem strange, but this dynamic generally holds true in all organizations, from your neighborhood elementary school up to a nation state. You could call it the iron law of institutions: The people running things almost always would rather be firmly in charge of a weaker institution than be part of a stronger institution in which their power can be challenged.

America’s leaders have put a lot of work into this maleficent project over the past 50 years, and their success can be measured in just two startling statistics. First, when the federal minimum wage was established in 1938, it was worth about $5.30 in today’s dollars. Then for the next three decades, it went up hand in hand with the U.S. economy’s growing productivity until 1968, when it reached about $13 an hour. But since then, it’s not just stopped going up, it’s also fallen in real value to today’s $7.25. However, if it had continued going up with productivity, it would now be around $25. That means that a couple who both work minimum wage jobs would make $100,000 per year. We know from those first 30 years of the minimum wage that there’s no economic reason that that was impossible. It’s just been politically impossible.

Second, the ultra-establishment RAND Corporation recently calculated how much money the increasing income inequality over the past decades has cost most Americans. They looked at the U.S. income distribution in 1975, and then calculated how things would have played out over the next 43 years through 2018 if the income distribution had remained at 1975’s level. RAND’s finding: The bottom 90 percent would have taken home a cumulative $50 trillion more in pay. (And of course the top 10 percent would have taken home $50 trillion less.) Again, this was a political choice, not an economic necessity.

This is the background to the economic fallout from the coronavirus pandemic.

With the sudden eruption of Covid-19, all normal rules of American politics were suspended. The government was willing to spend truly gigantic amounts of money, particularly via the CARES Act in 2020 and the American Rescue Plan in 2021. Moreover, large tranches of that spending essentially just involved giving people money.

There are two possible stories about what happened next. This is where the Regmi and Stiglitz paper comes in.

U.S. economist and recipient of the Nobel Prize in economic sciences Joseph E. Stiglitz poses during a photo session in Paris on Sept. 15, 2022.

Photo: Joel Saget/AFP via Getty Images

The inflation of the past several years has genuinely been too high and too sustained for the well-being of the U.S. If it was caused by too much demand — i.e., the government showering Americans with cash that they then wantonly spent — this would demonstrate the dangers of profligacy. In that scenario, we should be grateful that those days of extended unemployment benefits and an expanded child tax credit are behind us. What we need now is for the Federal Reserve to continue to bludgeon the economy with interest rates hikes until unemployment goes up and Americans stop buying stuff. We can then move forward, having learned a bitter lesson about government interference in the workings of the free market.

But if the inflation was caused by insufficient supply — due to the pandemic and the war in Ukraine — the situation would look completely different. That would mean these years of higher inflation were largely unavoidable, and the firehose of government cash was the wisest possible response to an emergency. It would also indicate that regular people should consider how to best use the government’s extraordinary power to conjure up spending from thin air in future emergencies, and even non-emergencies.

Regmi and Stiglitz’s argument is well worth reading in full. But these are the highlights:

To begin with, they illustrate that there was no spike in aggregate demand over the past several years. Rather, the spending by Americans on personal consumption fell off a cliff at the beginning of the pandemic — and then rebounded to the pre-pandemic trend by mid-2021. But inflation began rising when demand was still significantly below trend. As they put it, “this simple comparison refutes the claim that excessive consumption was the central cause of excessive inflation that followed the post-pandemic recovery.” They therefore conclude that “the inflation we’ve experienced is not best understood as an excess of aggregate demand over aggregate potential supply. Rather, today’s inflation is the result of a series of microeconomic, industry-specific problems.”

They then examine these industry-specific problems. They point out that of the 7.7 percent year-over-year rate of inflation as of October 2022, almost 3 percentage points were due to increasing energy and food costs. Both sectors were strongly affected by Russia’s invasion of Ukraine, and the subsequent sanctions the U.S. placed on Russia.

Another significant source of inflation has been the car industry. In 2021, Regmi and Stiglitz say, 1.94 percentage points of inflation “were due to automobiles and car parts.” And “key to the car shortage,” they write, “was the lack of microchips, and the lack of microchips was because of a simple market failure: By and large, car manufacturers (except Tesla) had canceled chip orders at the onset of the pandemic.”

Corporate profiteering — i.e., companies using inflation as an excuse to raise prices faster than their costs — has also contributed to inflation. 

The paper further argues that corporate profiteering — i.e., companies using inflation as an excuse to raise prices faster than their costs — has also contributed to inflation. This is illustrated by the fact that “firms with the most market power drove the sharp increase in aggregate markups in 2021.”

Then there have been shifts in demand — e.g., a desire toward bigger houses as people work more from home, and away from office buildings. While overall demand hasn’t increased, this causes inflation in the higher-demand sectors.

So the picture, when all is said and done, is pretty clear. The scale of the government response to the pandemic was extraordinary: Regmi and Stiglitz say that the International Monetary Fund “estimated that the fiscal stimulus related to the pandemic was 25.5 percent of the total US GDP.” Yet this spending over the past two years contributed to inflation in only a modest way at most.

This in turn tells us that we should not fear using the government’s power in bad times to prevent the economy from collapsing, and should look carefully at the role it could play in good times. Of course, that’s a lesson we could have learned long ago. After World War II, everyone had seen with their own eyes how the Keynesianism of the war had pulled the world economy out of the Great Depression. As the Polish economist Michal Kalecki wrote at the time, we had discovered we could create a more or less permanent “synthetic boom”: high wages, high worker power, and low unemployment. But we have to choose to do so. We can still make that choice, but only if we understand the reality in front of us.

The post The Pandemic and War — Not Government Spending — Caused Inflation, According to Nobel Prize Winner appeared first on The Intercept.

Nick and Ciarán talk about that Reichsbürger coup attempt sweeping the headline. The "Racing Heinrich" is a bit, we think it's an elaborate art project by Rob.

RECOMMENDED PAST EPISODES ON THIS TOPIC
The Dog From Paw Patrol Joins the KSK
Nazis With Antragsformulare

HOW TO SUPPORT US:
https://www.patreon.com/cornerspaeti

HOW TO REACH US:
Corner Späti https://twitter.com/cornerspaeti
Julia https://twitter.com/KMarxiana
Rob https://twitter.com/leninkraft
Nick https://twitter.com/sternburgpapi
Uma https://twitter.com/umawrnkl
Ciarán https://twitter.com/CiaranDold

Support Corner Späti

WE'RE UNLOCKING THIS BECAUSE EVENTS

Julia, Nick and Ciaran try to breakdown the weird German phenomenon of the "Reichsbürger" movement. While they're kooky weirdos who believe Germany is an illegitimate state and print their own money and passports, they also are well armed and want to commit terror attacks.

HOW TO SUPPORT US:
https://www.patreon.com/cornerspaeti

HOW TO REACH US:
Corner Späti https://twitter.com/cornerspaeti
Julia https://twitter.com/KMarxiana
Rob https://twitter.com/leninkraft
Nick https://twitter.com/sternburgpapi
Uma https://twitter.com/umawrnkl
Ciarán https://twitter.com/CiaranDold

Support Corner Späti

Krystal, Saagar and friends discuss Tom Cotton vs. Kroger, Potential Senate Investigations, Rail Strike Updates, Military Vaccine Mandates, CVS lawsuit, Monopolies/National Parks, and the Homeless Epidemic.

To become a Breaking Points Premium Member and watch/listen to the show uncut and 1 hour early visit: https://breakingpoints.supercast.com/

To listen to Breaking Points as a podcast, check them out on Apple and Spotify

Apple: https://podcasts.apple.com/us/podcast/breaking-points-with-krystal-and-saagar/id1570045623

Spotify: https://open.spotify.com/show/4Kbsy61zJSzPxNZZ3PKbXl

Merch: https://breaking-points.myshopify.com/

Learn more about your ad choices. Visit megaphone.fm/adchoices

My last post mentioned the out-of-the-box experience of Denote is not so great, especially when having notes across many subdirectories. By default, Denote searches and creates notes from the root denote-directory. I need to know the exact file location before searching for a note. I need to move note from the root directory to subdirectories after creating the note.

The good news is that, these issues are now fixed and this post talks about how.

#1 Search by Filename in Subdirectories

By default, Denote searches only in the denote-directory. To find a note hidden inside a subdirectory, the user needs to know the path to that file and navigate there. Needing to know the exact directory to be able to search for a file is counter-intuitive.

However, thanks to a recent (2022-11-18) refactoring by nobiot@, it is no longer the case! The commit makes notes inside subdirectories searchable with common Denote commands like denote-open-or-create and denote-link-*. This greatly improves the out-of-the-box experience of Denote. Just sync Denote to a newer version including this new change!

#2 Full-Text Search in Subdirectories

By default, Denote does not come with any command for full-text search.

However, it could be done easily with consult-ripgrep.

(defun consult-denote-ripgrep ()
 "Search with ‘rg’ for files in denote-directory where the content matches a regexp."
 (interactive)
 (consult-ripgrep denote-directory ""))

#3 Create Notes in Subdirectories

By default, Denote creates notes in the root denote-directory. Every time after creating a note, I need to move them manually to the right subdirectory. This is cumbersome.

However, it turned out to be my fault of not reading the official documentation, which has already suggested two ways to select a specific directory when creating a new note:

1. includes subdirectory to denote-prompts, or
2. uses M-x denote-subdirectory.

I go with (1) because it works with all Denote commands that could create a note, like denote-open-or-create and denote-link-or-create. It also allows other customizations to denote-prompts. For example, I prefer setting “subdirectory” before “title” (reorder title and subdirectory) and updating keyword later with denote-keyword-* (remove keyword from denote-prompts).

(setq denote-prompts '(subdirectory title))

If you have too much subdirectories, use denote-excluded-directories-regexp to exclude some of them.

Comparing with Org-roam Again

With these tweaks, I can now use Denote as my primary note-taking package.

I mentioned a concern with the link type denote: in my last post. It turns out to be not quite concerning because it is very easy to migrate to or away from this link type with some elisp. I don’t use other features like org-roam-daily, org-roam-bibtex, or org-roam-ui at all. The last step before removing Org-roam is probably migrating the existing links.

Bringing you all a podcast with some important breaking news segments from today including the latest Twitter Files scandal and Democratic Senator Kyrsten Sinema defecting from the party!

To become a Breaking Points Premium Member and watch/listen to the show uncut and 1 hour early visit: https://breakingpoints.supercast.com/

To listen to Breaking Points as a podcast, check them out on Apple and Spotify

Apple: https://podcasts.apple.com/us/podcast/breaking-points-with-krystal-and-saagar/id1570045623

Spotify: https://open.spotify.com/show/4Kbsy61zJSzPxNZZ3PKbXl

Merch: https://breaking-points.myshopify.com/

Learn more about your ad choices. Visit megaphone.fm/adchoices

Hosted on Acast. See acast.com/privacy for more information.

Hosted on Acast. See acast.com/privacy for more information.

Melvyn Bragg and guests discuss one of the greatest changes in the history of life on Earth. Around 400 million years ago some of our ancestors, the fish, started to become a little more like humans. At the swampy margins between land and water, some fish were turning their fins into limbs, their swim bladders into lungs and developed necks and eventually they became tetrapods, the group to which we and all animals with backbones and limbs belong. After millions of years of this transition, these tetrapod descendants of fish were now ready to leave the water for a new life of walking on land, and with that came an explosion in the diversity of life on Earth.

The image above is a representation of Tiktaalik Roseae, a fish with some features of a tetrapod but not one yet, based on a fossil collected in the Canadian Arctic.

With

Emily Rayfield Professor of Palaeobiology at the University of Bristol

Michael Coates Chair and Professor of Organismal Biology and Anatomy at the University of Chicago

And

Steve Brusatte Professor of Palaeontology and Evolution at the University of Edinburgh

Producer: Simon Tillotson

Melvyn Bragg and guests discuss the celebrated British poet of World War One. Wilfred Owen (1893-1918) had published only a handful of poems when he was killed a week before the end of the war, but in later decades he became seen as the essential British war poet. His works such as Anthem for Doomed Youth, Strange Meeting and Dulce et Decorum Est went on to be inseparable from the memory of the war and its futility. However, while Owen is best known for his poetry of the trenches, his letters offer a more nuanced insight into him such as his pride in being an officer in charge of others and in being a soldier who fought alongside his comrades.

With

Jane Potter Reader in The School of Arts at Oxford Brookes University

Fran Brearton Professor of Modern Poetry at Queen’s University Belfast

And

Guy Cuthbertson Professor of British Literature and Culture at Liverpool Hope University

Producer: Simon Tillotson

Melvyn Bragg and guests discuss the rebellion that broke out in Jamaica on 11th October 1865 when Paul Bogle (1822-65) led a protest march from Stony Gut to the courthouse in nearby Morant Bay. There were many grounds for grievance that day and soon anger turned to bloodshed. Although the British had abolished slavery 30 years before, the plantation owners were still dominant and the conditions for the majority of people on Jamaica were poor. The British governor suppressed this rebellion brutally and soon people in Jamaica lost what right they had to rule themselves. Some in Britain, like Charles Dickens, supported the governor's actions while others, like Charles Darwin, wanted him tried for murder.

The image above is from a Jamaican $2 banknote, printed after Paul Bogle became a National Hero in 1969.

With

Matthew J Smith Professor of History and Director of the Centre for the Study of the Legacies of British Slavery at University College London

Diana Paton The William Robertson Professor of History at the University of Edinburgh

And

Lawrence Goldman Emeritus Fellow in History at St Peter’s College, University of Oxford

Producer: Simon Tillotson