Pipenv is a packaging tool for Python that solves some common problems associated with the typical workflow using pip, virtualenv, and the good old requirements.txt.

In addition to addressing some common issues, it consolidates and simplifies the development process to a single command line tool.

This guide will go over what problems Pipenv solves and how to manage your Python dependencies with Pipenv. Additionally, it will cover how Pipenv fits in with previous methods for package distribution.

Problems that Pipenv Solves

To understand the benefits of Pipenv, it’s important to walk through the current methods for packaging and dependency management in Python.

Let’s start with a typical situation of handling third-party packages. We’ll then build our way towards deploying a complete Python application.

Dependency Management with requirements.txt

Imagine you’re working on a Python project that uses a third-party package like flask. You’ll need to specify that requirement so that other developers and automated systems can run your application.

So you decide to include the flask dependency in a requirements.txt file:

flask

Great, everything works fine locally, and after hacking away on your app for a while, you decide to move it to production. Here’s where things get a little scary…

The above requirements.txt file doesn’t specify which version of flask to use. In this case, pip install -r requirements.txt will install the latest version by default. This is okay unless there are interface or behavior changes in the newest version that break our application.

For the sake of this example, let’s say that a new version of flask got released. However, it isn’t backward compatible with the version you used during development.

Now, let’s say you deploy your application to production and do a pip install -r requirements.txt. Pip gets the latest, not-backward-compatible version of flask, and just like that, your application breaks… in production.

“But hey, it worked on my machine!”—I’ve been there myself, and it’s not a great feeling.

At this point, you know that the version of flask you used during development worked fine. So, to fix things, you try to be a little more specific in your requirements.txt. You add a version specifier to the flask dependency. This is also called pinning a dependency:

flask==0.12.1

Pinning the flask dependency to a specific version ensures that a pip install -r requirements.txt sets up the exact version of flask you used during development. But does it really?

Keep in mind that flask itself has dependencies as well (which pip installs automatically). However, flask itself doesn’t specify exact versions for its dependencies. For example, it allows any version of Werkzeug>=0.14.

Again, for the sake of this example, let’s say a new version of Werkzeug got released, but it introduces a show-stopper bug to your application.

When you do pip install -r requirements.txt in production this time, you will get flask==0.12.1 since you’ve pinned that requirement. However, unfortunately, you’ll get the latest, buggy version of Werkzeug. Again, the product breaks in production.

The real issue here is that the build isn’t deterministic. What I mean by that is that, given the same input (the requirements.txt file), pip doesn’t always produce the same environment. At the moment, you can’t easily replicate the exact environment you have on your development machine in production.

The typical solution to this problem is to use pip freeze. This command allows you to get exact versions for all 3rd party libraries currently installed, including the sub-dependencies pip installed automatically. So you can freeze everything in development to ensure that you have the same environment in production.

Executing pip freeze results in pinned dependencies you can add to a requirements.txt:

click==6.7
Flask==0.12.1
itsdangerous==0.24
Jinja2==2.10
MarkupSafe==1.0
Werkzeug==0.14.1

With these pinned dependencies, you can ensure that the packages installed in your production environment match those in your development environment exactly, so your product doesn’t unexpectedly break. This “solution,” unfortunately, leads to a whole new set of problems.

Now that you’ve specified the exact versions of every third-party package, you are responsible for keeping these versions up to date, even though they’re sub-dependencies of flask. What if there’s a security hole discovered in Werkzeug==0.14.1 that the package maintainers immediately patched in Werkzeug==0.14.2? You really need to update to Werkzeug==0.14.2 to avoid any security issues arising from the earlier, unpatched version of Werkzeug.

First, you need to be aware that there’s an issue with the version you have. Then, you need to get the new version in your production environment before someone exploits the security hole. So, you have to change your requirements.txt manually to specify the new version Werkzeug==0.14.2. As you can see in this situation, the responsibility of staying up to date with necessary updates falls on you.

The truth is that you really don’t care what version of Werkzeug gets installed as long as it doesn’t break your code. In fact, you probably want the latest version to ensure that you’re getting bug fixes, security patches, new features, more optimization, and so on.

The real question is: “How do you allow for deterministic builds for your Python project without gaining the responsibility of updating versions of sub-dependencies?”

Spoiler alert: The easy answer is using Pipenv.

Development of Projects with Different Dependencies

Let’s switch gears a bit to talk about another common issue that arises when you’re working on multiple projects. Imagine that ProjectA needs django==1.9, but ProjectB needs django==1.10.

By default, Python tries to store all your third-party packages in a system-wide location. This means that every time you want to switch between ProjectA and ProjectB, you have to make sure the right version of django is installed. This makes switching between projects painful because you have to uninstall and reinstall packages to meet the requirements for each project.

The standard solution is to use a virtual environment that has its own Python executable and third-party package storage. That way, ProjectA and ProjectB are adequately separated. Now you can easily switch between projects since they’re not sharing the same package storage location. PackageA can have whatever version of django it needs in its own environment, and PackageB can have what it needs totally separate. A very common tool for this is virtualenv (or venv in Python 3).

Pipenv has virtual environment management built in so that you have a single tool for your package management.

Dependency Resolution

What do I mean by dependency resolution? Let’s say you’ve got a requirements.txt file that looks something like this:

package_a
package_b

Let’s say package_a has a sub-dependency package_c, and package_a requires a specific version of this package: package_c>=1.0. In turn, package_b has the same sub-dependency but needs package_c<=2.0.

Ideally, when you try to install package_a and package_b, the installation tool would look at the requirements for package_c (being >=1.0 and <=2.0) and select a version that fulfills those requirements. You’d hope that the tool resolves the dependencies so that your program works in the end. This is what I mean by “dependency resolution.”

Unfortunately, pip itself doesn’t have real dependency resolution at the moment, but there’s an open issue to support it.

The way pip would handle the above scenario is as follows:

1. It installs package_a and looks for a version of package_c that fulfills the first requirement (package_c>=1.0).

2. Pip then installs the latest version of package_c to fulfill that requirement. Let’s say the latest version of package_c is 3.1.

This is where the trouble (potentially) starts.

If the version of package_c selected by pip doesn’t fit future requirements (such as package_b needing package_c<=2.0), the installation will fail.

The “solution” to this problem is to specify the range required for the sub-dependency (package_c) in the requirements.txt file. That way, pip can resolve this conflict and install a package that meets those requirements:

package_c>=1.0,<=2.0
package_a
package_b

Just like before though, you’re now concerning yourself directly with sub-dependencies (package_c). The issue with this is that if package_a changes their requirement without you knowing, the requirements you specified (package_c>=1.0,<=2.0) may no longer be acceptable, and installation may fail… again. The real problem is that once again, you’re responsible for staying up to date with requirements of sub-dependencies.

Ideally, your installation tool would be smart enough to install packages that meet all the requirements without you explicitly specifying sub-dependency versions.

Pipenv Introduction

Now that we’ve addressed the problems, let’s see how Pipenv solves them.

First, let’s install it:

$ pip install pipenv

Once you’ve done that, you can effectively forget about pip since Pipenv essentially acts as a replacement. It also introduces two new files, the Pipfile (which is meant to replace requirements.txt) and the Pipfile.lock (which enables deterministic builds).

Pipenv uses pip and virtualenv under the hood but simplifies their usage with a single command line interface.

Example Usage

Let’s start over with creating your awesome Python application. First, spawn a shell in a virtual environment to isolate the development of this app:

$ pipenv shell

This will create a virtual environment if one doesn’t already exist. Pipenv creates all your virtual environments in a default location. If you want to change Pipenv’s default behavior, there are some environmental variables for configuration.

You can force the creation of a Python 2 or 3 environment with the arguments --two and --three respectively. Otherwise, Pipenv will use whatever default virtualenv finds.

Sidenote: If you require a more specific version of Python, you can provide a --python argument with the version you require. For example: --python 3.6

Now you can install the 3rd party package you need, flask. Oh, but you know that you need version 0.12.1 and not the latest version, so go ahead and be specific:

$ pipenv install flask==0.12.1

You should see something like the following in your terminal:

Adding flask==0.12.1 to Pipfile's [packages]…
Pipfile.lock not found, creating…

You’ll notice that two files get created, a Pipfile and Pipfile.lock. We’ll take a closer look at these in a second. Let’s install another 3rd party package, numpy, for some number-crunching. You don’t need a specific version so don’t specify one:

$ pipenv install numpy

If you want to install something directly from a version control system (VCS), you can! You specify the locations similarly to how you’d do so with pip. For example, to install the requests library from version control, do the following:

$ pipenv install -e git+https://github.com/requests/requests.git#egg=requests

Note the -e argument above to make the installation editable. Currently, this is required for Pipenv to do sub-dependency resolution.

Let’s say you also have some unit tests for this awesome application, and you want to use pytest for running them. You don’t need pytest in production so you can specify that this dependency is only for development with the --dev argument:

$ pipenv install pytest --dev

Providing the --dev argument will put the dependency in a special [dev-packages] location in the Pipfile. These development packages only get installed if you specify the --dev argument with pipenv install.

The different sections separate dependencies needed only for development from ones needed for the base code to actually work. Typically, this would be accomplished with additional requirements files like dev-requirements.txt or test-requirements.txt. Now, everything is consolidated in a single Pipfile under different sections.

Okay, so let’s say you’ve got everything working in your local development environment and you’re ready to push it to production. To do that, you need to lock your environment so you can ensure you have the same one in production:

$ pipenv lock

This will create/update your Pipfile.lock, which you’ll never need to (and are never meant to) edit manually. You should always use the generated file.

Now, once you get your code and Pipfile.lock in your production environment, you should install the last successful environment recorded:

$ pipenv install --ignore-pipfile

This tells Pipenv to ignore the Pipfile for installation and use what’s in the Pipfile.lock. Given this Pipfile.lock, Pipenv will create the exact same environment you had when you ran pipenv lock, sub-dependencies and all.

The lock file enables deterministic builds by taking a snapshot of all the versions of packages in an environment (similar to the result of a pip freeze).

Now let’s say another developer wants to make some additions to your code. In this situation, they would get the code, including the Pipfile, and use this command:

$ pipenv install --dev

This installs all the dependencies needed for development, which includes both the regular dependencies and those you specified with the --dev argument during install.

When an exact version isn’t specified in the Pipfile, the install command gives the opportunity for dependencies (and sub-dependencies) to update their versions.

This is an important note because it solves some of the previous problems we discussed. To demonstrate, let’s say a new version of one of your dependencies becomes available. Because you don’t need a specific version of this dependency, you don’t specify an exact version in the Pipfile. When you pipenv install, the new version of the dependency will be installed in your development environment.

Now you make your changes to the code and run some tests to verify everything is still working as expected. (You do have unit tests, right?) Now, just as before, you lock your environment with pipenv lock, and an updated Pipfile.lock will be generated with the new version of the dependency. Just as before, you can replicate this new environment in production with the lock file.

As you can see from this scenario, you no longer have to force exact versions you don’t truly need to ensure your development and production environments are the same. You also don’t need to stay on top of updating sub-dependencies you “don’t care about.” This workflow with Pipenv, combined with your excellent testing, fixes the issues of manually doing all your dependency management.

Pipenv’s Dependency Resolution Approach

Pipenv will attempt to install sub-dependencies that satisfy all the requirements from your core dependencies. However, if there are conflicting dependencies (package_a needs package_c>=1.0, but package_b needs package_c<1.0), Pipenv will not be able to create a lock file and wil output an error like the following:

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches package_c>=1.0,package_c<1.0

As the warning says, you can also show a dependency graph to understand your top-level dependencies and their sub-dependencies:

$ pipenv graph

This command will print out a tree-like structure showing your dependencies. Here’s an example:

Flask==0.12.1
  - click [required: >=2.0, installed: 6.7]
  - itsdangerous [required: >=0.21, installed: 0.24]
  - Jinja2 [required: >=2.4, installed: 2.10]
    - MarkupSafe [required: >=0.23, installed: 1.0]
  - Werkzeug [required: >=0.7, installed: 0.14.1]
numpy==1.14.1
pytest==3.4.1
  - attrs [required: >=17.2.0, installed: 17.4.0]
  - funcsigs [required: Any, installed: 1.0.2]
  - pluggy [required: <0.7,>=0.5, installed: 0.6.0]
  - py [required: >=1.5.0, installed: 1.5.2]
  - setuptools [required: Any, installed: 38.5.1]
  - six [required: >=1.10.0, installed: 1.11.0]
requests==2.18.4
  - certifi [required: >=2017.4.17, installed: 2018.1.18]
  - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
  - idna [required: >=2.5,<2.7, installed: 2.6]
  - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]

From the output of pipenv graph, you can see the top-level dependencies we installed previously (Flask, numpy, pytest, and requests), and underneath you can see the packages they depend on.

Additionally, you can reverse the tree to show the sub-dependencies with the parent that requires it:

$ pipenv graph --reverse

This reversed tree may be more useful when you are trying to figure out conflicting sub-dependencies.

The Pipfile

Pipfile intends to replace requirements.txt. Pipenv is currently the reference implementation for using Pipfile. It seems very likely that pip itself will be able to handle these files. Also, it’s worth noting that Pipenv is even the official package management tool recommended by Python itself.

The syntax for the Pipfile is TOML, and the file is separated into sections. [dev-packages] for development-only packages, [packages] for minimally required packages, and [requires] for other requirements like a specific version of Python. See an example file below:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]
pytest = "*"

[packages]
flask = "==0.12.1"
numpy = "*"
requests = {git = "https://github.com/requests/requests.git", editable = true}

[requires]
python_version = "3.6"

Ideally, you shouldn’t have any sub-dependencies in your Pipfile. What I mean by that is you should only include the packages you actually import and use. No need to keep chardet in your Pipfile just because it’s a sub-dependency of requests. (Pipenv will install it automatically.) The Pipfile should convey the top-level dependencies your package requires.

The Pipfile.lock

This file enables deterministic builds by specifying the exact requirements for reproducing an environment. It contains exact versions for packages and hashes to support more secure verification, which pip itself now supports as well. An example file might look like the following. Note that the syntax for this file is JSON and that I’ve excluded parts of the file with ...:

{
    "_meta": {
        ...
    },
    "default": {
        "flask": {
            "hashes": [
                "sha256:6c3130c8927109a08225993e4e503de4ac4f2678678ae211b33b519c622a7242",
                "sha256:9dce4b6bfbb5b062181d3f7da8f727ff70c1156cbb4024351eafd426deb5fb88"
            ],
            "version": "==0.12.1"
        },
        "requests": {
            "editable": true,
            "git": "https://github.com/requests/requests.git",
            "ref": "4ea09e49f7d518d365e7c6f7ff6ed9ca70d6ec2e"
        },
        "werkzeug": {
            "hashes": [
                "sha256:d5da73735293558eb1651ee2fddc4d0dedcfa06538b8813a2e20011583c9e49b",
                "sha256:c3fd7a7d41976d9f44db327260e263132466836cef6f91512889ed60ad26557c"
            ],
            "version": "==0.14.1"
        }
        ...
    },
    "develop": {
        "pytest": {
            "hashes": [
                "sha256:8970e25181e15ab14ae895599a0a0e0ade7d1f1c4c8ca1072ce16f25526a184d",
                "sha256:9ddcb879c8cc859d2540204b5399011f842e5e8823674bf429f70ada281b3cc6"
            ],
            "version": "==3.4.1"
        },
        ...
    }
}

Note the exact version specified for every dependency. Even the sub-dependencies like werkzeug that aren’t in our Pipfile appear in this Pipfile.lock. The hashes are used to ensure you’re retrieving the same package as you did in development.

It’s worth noting again that you should never change this file by hand. It is meant to be generated with pipenv lock.

Pipenv Extra Features

Open a third-party package in your default editor with the following command:

$ pipenv open flask

This will open the flask package in the default editor, or you can specify a program with an EDITOR environmental variable. For example, I use Sublime Text, so I just set EDITOR=subl. This makes it super simple to dig into the internals of a package you’re using.

You can run a command in the virtual environment without launching a shell:

$ pipenv run <insert command here>

Check for security vulnerabilities (and PEP 508 requirements) in your environment:

$ pipenv check

Now, let’s say you no longer need a package. You can uninstall it:

$ pipenv uninstall numpy

Additionally, let’s say you want to completely wipe all the installed packages from your virtual environment:

$ pipenv uninstall --all

You can replace --all with --all-dev to just remove dev packages.

Pipenv supports the automatic loading of environmental variables when a .env file exists in the top-level directory. That way, when you pipenv shell to open the virtual environment, it loads your environmental variables from the file. The .env file just contains key-value pairs:

SOME_ENV_CONFIG=some_value
SOME_OTHER_ENV_CONFIG=some_other_value

Finally, here are some quick commands to find out where stuff is. How to find out where your virtual environment is:

$ pipenv --venv

How to find out where your project home is:

$ pipenv --where

Package Distribution

You may be asking how this all works if you intend to distribute your code as a package.

Yes, I need to distribute my code as a package

How does Pipenv work with setup.py files?

There are a lot of nuances to that question. First, a setup.py file is necessary when you’re using setuptools as your build/distribution system. This has been the de facto standard for a while now, but recent changes have made the use of setuptools optional.

This means that projects like flit can use the new pyproject.toml to specify a different build system that doesn’t require a setup.py.

All that being said, for the near future setuptools and an accompanying setup.py will still be the default choice for many people.

Here’s a recommended workflow for when you are using a setup.py as a way to distribute your package:

• setup.py
• install_requires keyword should include whatever the package “minimally needs to run correctly.”
• Pipfile
• Represents the concrete requirements for your package
• Pull the minimally required dependencies from setup.py by installing your package using Pipenv:
  • Use pipenv install '-e .'
  • That will result in a line in your Pipfile that looks something like "e1839a8" = {path = ".", editable = true}.
• Pipfile.lock
• Details for a reproducible environment generated from pipenv lock

To clarify, put your minimum requirements in setup.py instead of directly with pipenv install. Then use the pipenv install '-e .' command to install your package as editable. This gets all the requirements from setup.py into your environment. Then you can use pipenv lock to get a reproducible environment.

I don’t need to distribute my code as a package

Great! If you are developing an application that isn’t meant to be distributed or installed (a personal website, a desktop application, a game, or similar), you don’t really need a setup.py.

In this situation, you could use Pipfile/Pipfile.lock combo for managing your dependencies with the flow described previously to deploy a reproducible environment in production.

I already have a requirements.txt. How do I convert to a Pipfile?

If you run pipenv install it should automatically detect the requirements.txt and convert it to a Pipfile, outputting something like the following:

requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did.
We recommend updating your Pipfile to specify the "*" version, instead.

Take note of the above warning.

If you have pinned exact versions in your requirements.txt file, you’ll probably want to change your Pipfile to only specify exact versions you truly require. This will allow you to gain the real benefits of transitioning. For example, let’s say you have the following but really don’t need that exact version of numpy:

[packages]
numpy = "==1.14.1"

If you don’t have any specific version requirements for your dependencies, you can use the wildcard character * to tell Pipenv that any version can be installed:

[packages]
numpy = "*"

If you feel nervous about allowing any version with the *, it’s typically a safe bet to specify greater than or equal to the version you’re already on so you can still take advantage of new versions:

[packages]
numpy = ">=1.14.1"

Of course, staying up to date with new releases also means you’re responsible for ensuring your code still functions as expected when packages change. This means a test suite is essential to this whole Pipenv flow if you want to ensure functioning releases of your code.

You allow packages to update, run your tests, ensure they all pass, lock your environment, and then you can rest easy knowing that you haven’t introduced breaking changes. If things do break because of a dependency, you’ve got some regression tests to write and potentially some more restrictions on versions of dependencies.

For example, if numpy==1.15 gets installed after running pipenv install and it breaks your code, which you hopefully either notice during development or during your tests, you have a couple options:

1. Update your code to function with the new version of the dependency.

   If backward compatibility with previous versions of the dependency isn’t possible, you’ll also need to bump your required version in your Pipfile:

   [packages]
   numpy = ">=1.15"
   

2. Restrict the version of the dependency in the Pipfile to be < the version that just broke your code:

   [packages]
   numpy = ">=1.14.1,<1.15"
   

Option 1 is preferred as it ensures that your code is using the most up-to-date dependencies. However, Option 2 takes less time and doesn’t require code changes, just restrictions on dependencies.

You can also install from requirement files with the same -r argument pip takes:

$ pipenv install -r requirements.txt

If you have a dev-requirements.txt or something similar, you can add those to the Pipfile as well. Just add the --dev argument so it gets put in the right section:

$ pipenv install -r dev-requirements.txt --dev

Additionally, you can go the other way and generate requirements files from a Pipfile:

$ pipenv lock -r > requirements.txt
$ pipenv lock -r -d > dev-requirements.txt

What’s next?

It appears to me that a natural progression for the Python ecosystem would be a build system that uses the Pipfile to install the minimally required dependencies when retrieving and building a package from a package index (like PyPI). It is important to note again that the Pipfile design specification is still in development, and Pipenv is just a reference implementation.

That being said, I could see a future where the install_requires section of setup.py doesn’t exist, and the Pipfile is referenced for minimal requirements instead. Or the setup.py is gone entirely, and you get metadata and other information in a different manner, still using the Pipfile to get the necessary dependencies.

Is Pipenv worth checking out?

Definitely. Even if it’s just as a way to consolidate the tools you already use (pip & virtualenv) into a single interface. However, it’s much more than that. With the addition of the Pipfile, you only specify the dependencies you truly need.

You no longer have the headache of managing the versions of everything yourself just to ensure you can replicate your development environment. With the Pipfile.lock, you can develop with peace of mind knowing that you can exactly reproduce your environment anywhere.

In addition to all that, it seems very likely that the Pipfile format will get adopted and supported by official Python tools like pip, so it’d be beneficial to be ahead of the game. Oh, and make sure you’re updating all your code to Python 3 as well: 2020 is coming up fast.

References, further reading, interesting discussions, and so forth

• Official Pipenv documentation
• Official Pipfile Project
• Issue addressing install_requires in regards to Pipfile
• More discussion on setup.py vs Pipfile
• Post talking about PEP 518
• Post on Python packaging
• Comment suggesting Pipenv usage

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

AUTHOR.

(Photo: 

U.S. Army soldiers assigned to the 24th Infantry Regiment move up to the firing line in Korea, July 18, 1950. U.S. Army photo)

http://JohnBatchelorShow.com/contact

http://JohnBatchelorShow.com/schedules

Twitter: @BatchelorShow

Winning Korea is winning the Silk Road: Empires of the Silk Road: 2 of 2: A History of Central Eurasia from the Bronze Age to the Present 1by Christopher I. Beckwith

AUTHOR.

(Photo:

English: Vought F4U-4 Corsair fighters of U.S. Navy fighter squadrons VF-114 Executioners (V-4XX) and VF-113 Stingers (V-3XX, on the catapult), assigned to Carrier Air Group 11 (CVG-11), on the aircraft carrier USS Philippine Sea (CV-47) during the Korean War in 1951/52. Another Essex-class carrier of Task Force 77 is visible in the background.

Date 1951/52

Source U.S. DefenseImagery [1] photo VIRIN: HA-SN-98-06975

Author USN)

http://JohnBatchelorShow.com/contact

http://JohnBatchelorShow.com/schedules

Twitter: @BatchelorShow

Winning Korea is winning the Silk Road: Empires of the Silk Road: 1 of 2: A History of Central Eurasia from the Bronze Age to the Present 1by Christopher I. Beckwith

Julian Assange has been barred from communicating with the outside world for more than three weeks. On March 27, the government of Ecuador blocked Assange’s internet access and barred him from receiving visitors other than his lawyers. Assange has been in the Ecuadorian embassy in London since 2012, when Ecuador granted him asylum due to fears that his extradition to Sweden as part of a sexual assault investigation would result in his being sent to the U.S. for prosecution for his work with WikiLeaks. In January of this year, Assange formally became a citizen of Ecuador.

As a result of Ecuador’s recent actions, Assange — long a prolific commentator on political debates around the world — has been silenced for more than three weeks, by a country that originally granted him political asylum and of which he is now a citizen. While Ecuador was willing to defy Western dictates to hand over Assange under the presidency of Rafael Correa — who was fiercely protective of Ecuadorian sovereignty even if it meant disobeying Western powers — his successor, Lenín Moreno, has proven himself far more subservient, and that mentality — along with Moreno’s increasingly bitter feud with Correa — are major factors in the Ecuadorian government’s newly hostile treatment of Assange.

Yet many of the recent media claims about Assange that have caused this standoff — which have centered on the alleged role of Russia in the internal Spanish conflict over Catalan independence — range from highly dubious to demonstrably false. The campaign to depict Catalan unrest as a plot fueled by the Kremlin, Assange and even Edward Snowden have largely come from fraudulent assertions in the Spanish daily El País and  highly dubious data claims from the so-called Hamilton 68 dashboard. The consequences of these false and misleading claims — this actual “fake news” — have been multifaceted and severe, not just for Assange, but for diplomatic relations among multiple countries.

The Guardian reported last week that doctors who recently visited Assange concluded his health condition has become “dangerous.” The journalist Stefania Maurizi of La Repubblica yesterday confirmed that Assange “is still in the Ecuadorian Embassy in London and unable to access the internet and to receive visitors,” while the official WikiLeaks account provided further details about the restrictions Assange faces:

Ordinarily, Western commentators would be lining up to denounce a country like Ecuador for blocking the communications and internet access of one of its own citizens. But because the person silenced here is Assange, whom they hate, their heartfelt devotion to the sacred principles of free speech and a free press vanish.

(When Ecuador first granted asylum to Assange, both the Ecuadorian government and Assange’s lawyers have always said that Assange would board the next flight to Stockholm if the Swedish government gave assurances it would not extradite him to the U.S. Although Swedish prosecutors last year dropped the sex assault investigation into Assange, Trump CIA Director Mike Pompeo has vowed to do everything possible to destroy WikiLeaks and prevent it from publishing further, while the U.K. government — an ally of the Trump administration — has vowed to arrest Assange on bail charges if he leaves the embassy.)

Evidence has now emerged that the cutting off of Assange’s communications with the outside world is the byproduct of serious diplomatic pressure being applied to the new Ecuadorian president, pressure that may very well lead, perhaps imminently, to Assange being expelled from the embassy altogether. The pressure is coming from the Spanish government in Madrid and its NATO allies, furious that Assange has expressed opposition to some of the repressive measures used to try to crush activists in support of Catalan independence.

The day after blocking Assange’s communications with the outside world, Ecuador issued a statement alleging that Assange’s public statements are “putting at risk” Ecuador’s relations with other states. The Ecuadorian government has previously expressed dissatisfaction with some of Assange’s political activities and statements, but the breaking point appears to have been a series of tweets from Assange about the arrest in Germany earlier this month of former President of Catalonia Carles Puigdemont.

Beginning in September, Assange had been tweeting regularly about the referendum for independence in Catalonia. Back then, Ecuador released a statement criticizing these tweets and emphasizing that “Ecuadorian authorities have reiterated to Mr. Assange his obligation not to make statements or activities that could affect Ecuador’s international relations.”

But why did these tweets about Catalonia, of all of Assange’s tweets about politics in other countries and the role he played in the 2016 U.S. election, lead — after five years — to such a response? And why now? And why and how did the West succeed in convincing so many of its citizens that the movement for Catalan independence — which has been a source of internal conflict in Spain for years — was now suddenly fueled by a Kremlin plot with Putin as the puppet master?

The answers provide a vivid example of how claims about “fake news,” Western propaganda, and disinformation can be used as a tactic for political manipulation. The circumstances leading to recent events extend beyond Julian Assange and understanding them requires background on political pressures in Spain, Ecuador, the United States, and the United Kingdom, and how these intersect with Assange’s case.

The tensions between Ecuador and Assange center on the debate in Spain over Catalan independence. On October 1, 2017, the autonomous region of Catalonia held a referendum for independence. The Spanish government declared this referendum illegal. Protests and arrests of Catalan activists ensued, as well as the seizure of ballots and raids on polling stations by the government in Madrid.

In the midst of this crisis, former Spanish Prime Minister Felipe González reportedly requested that Spain’s most powerful media conglomerate, Grupo PRISA, which owns El País, “offer a firm response” to the independence movement in Catalonia. The media corporation complied, devoting its full resources to opposing Catalan secession.

El País, days later, began depicting Catalan activists as a tool of the Kremlin. The paper published an article alleging that not only Assange, but also Edward Snowden, were helping Russian propaganda networks spread “fake news” about Catalonia. El País repeated these claims in subsequent stories, which were echoed in reports from other anti-separatist organizations, such as the Spanish think tank Elcano Royal Institute, Atlantic Council’s Digital Forensics Research Lab, and NATO’s StratCom.

El País, Sept. 25, 2017

Once El País endorsed these fantastical allegations — that Assange and Snowden were helping to lead a Kremlin campaign to promote Catalan separatism — they were cited by or brought before legislative bodies around the world, in Spain, the U.K., and the U.S. But while these accusations are being taken seriously, they — like many claims about “fake news” and foreign online propaganda campaigns — are not being critically scrutinized or journalistically verified, and have little evidentiary support.

When pressed, many of those advancing these claims admit they have no definitive evidence of Russian government interference during the referendum in Catalonia, capable of citing only what they regard as biased reporting from RT and Sputnik. This exchange, from a House of Commons hearing on “fake news,” illustrates this point:

Yet — as they do with most Western problems these days — they continue to depict the conflict in terms of Russian propaganda because, they themselves acknowledge, they cannot comprehend the tensions in Spain except as a Kremlin operation. As Mira Milosevich-Juaristi from Elcano said when testifying before the Comisión Mixta de Seguridad Nacional in Spain: “The complexity and combination and coordination, which all occur at the same time, need an actor either governmental or near the government to coordinate it.”

Accusations that online support for Catalan independence was a Russian plot led by Assange and Snowden became a widespread belief. From the start, it was based overwhelmingly on a crude “dashboard” calling itself “Hamilton 68″ that is maintained by the Alliance for Securing Democracy. The “dashboard” purports to track “the activities of 600 Twitter accounts linked to Russian influence efforts online.”

That Catalonia was trending among the supposedly pro-Kremlin accounts monitored by the dashboard was offered by El País as what the paper called “definitive proof” — definitive proof — “that those who mobilize the army of pro-Russian bots have chosen to focus on the Catalan independence movement.”

But from its inception, the dubiousness of Hamilton 68 was self-evident. As The Intercept reported when the group was first formed, it was a brand-new foreign policy advocacy organization created by the exact people with the worst records of lying and militarism in Washington: Bill Kristol, former CIA officials, GOP hawks, and Democratic Party neocons.

Even worse, Hamilton 68 was, and remains, incredibly opaque about its methodology, refusing even to identify which accounts they designate as “promoting Russian influence online.” These marked accounts not only include “accounts that clearly state they are pro-Russian or affiliated with the Russian government,” but also “accounts run by people around the world who amplify pro-Russian themes either knowingly or unknowingly” (which often includes any dissent from the U.S. foreign policy orthodoxies endorsed by the neocons and CIA officials who created the group, now branded as “pro-Russian”).

Despite all of these glaring reasons for skepticism, U.S. media outlets repeatedly ingested the claims of this brand-new, sketchy group about what “Russian bots” are saying without an iota of critical thought or questioning, constructing one headline after the next based exclusively on the claims of this murky, shady group. As Rolling Stone’s Matt Taibbi recently documented, “More and more often now, the site’s pronouncements turn into front-page headlines.”

But in recent months, the credibility of Hamilton 68 has been widely challenged. Journalists and researchers have identified numerous inaccurate stories that were based on Hamilton 68 data, examined the involvement of highly ideological actors in the development of the tool, and questioned the “secret methodology” of Hamilton 68 and specifically its bizarre refusal to disclose the list of 600 accounts on which it bases its data. Some additionally note that the narrative about Russian bots and trolls is increasingly used as a tool to discredit a wide variety of legitimate political movements around the world.

Often, these media stories based on Hamilton 68, that fuel hysteria over Russian control of the west, are founded upon a poor understanding of what can and cannot be affirmed based on the data. The U.S. media’s abuse of this data finally caused even one of the creators of Hamilton 68 to express frustration with these inaccurate conclusions based on the most superficial understanding of the data, saying: “It’s somewhat frustrating because sometimes we have people make claims about it or whatever — we’re like, that’s not what it says, go back and look at it.”

Misunderstanding social media analytic tools is a common problem in fake news reporting and can lead to erroneous conclusions with real political consequences. One illustrative example is the claim by El País – in its seminal article depicting Catalan independence as fueled by the Kremlin — that “a detailed analysis of 5,000 of Assange’s followers on Twitter provided by TwitterAudit, reveals that 59% are false profiles.”

El País’s claim is a demonstrable, obvious fraud. This assertion was entirely inaccurate because the data was from an inactive account with no tweets. Assange created a personal account years ago, but only started using it to tweet for the first time on February 14, 2017. But the Twitter Audit data used by El País to make this extraordinary claim — that Assange’s following is mostly composed of bots — is from February 2014, three years before anything was tweeted from the account.

To put it mildly, the archaic data used by El Pais regarding Assange’s account is wildly inaccurate for claims about his current followers. After reassessing @JulianAssange’s followers on November 24, 2017, Twitter Audit now shows that 92% of his followers are real.

Twitter Audit still claims that 8% of Assange’s followers are bots or otherwise fake, but this is relatively low considering that recent scientific studies estimate that “between 9% and 15% of active Twitter accounts are bots.”

It is also low relative to the results for other accounts with many followers. For example, Twitter Audit estimates that 25% of the @el_pais followers, 17% of @BarackObama’s followers, 27% of @RealDonaldTrump’s followers, and 48% of @EmmanuelMacron’s followers (as of nine months ago) may be fake accounts. Social media tools like Twitter Audit generally only provide rough heuristics, which are meaningless in isolation.

Correctly interpreting the results of social media analytics tools requires not only closely examining the data and understanding the limitations of the tools, but also comparing to known benchmarks from scientific studies and controls.

One example of this dubious methodological approach is the claim from both El País and DFRLab that a particular tweet by Julian Assange spread suspiciously quickly. On September 15, Assange tweeted: “I ask everyone to support Catalonia’s right to self-determination. Spain cannot be permitted to normalize repressive acts to stop the vote.”

El País argued, in the excerpt provided above, that “in the case of the tweet from Assange, as with many of his messages on the social media platform, it received 2,000 retweets in an hour and obtained its maximum reach – 12,000 retweets, in less than a day. The fact that the tweet went viral so quickly is evidence of the intervention of bots, or false social media profiles, programmed simply to automatically echo certain messages.”

The claim that retweet rates should gradually accelerate over time may make intuitive sense, but neither El País nor DFRLab provided any citations to research on these dynamics. In fact, at least one study has found that these supposedly intuitive hypotheses about retweet rates are wrong, and instead “half of retweeting occurs within an hour, and 75% under a day.” In the case of this particular tweet from Assange, only one-sixth of the retweets occurred in the first hour.

The overall spread of the tweet is also normal relative to Assange’s other tweets. Each of Julian Assange’s tweets between August 1 and December 12, 2017 was seen by 232,249.63 people on average. The specific tweet that El País scrutinized received 761,410 impressions (that is, this particular tweet was shown to other Twitter users 761,410 times). This is a bit higher than normal, but not disproportionately so; Assange’s most popular tweets regularly receive 3 or 4 million impressions, 4 to 6 times as many as the tweet that El País said must have been amplified by bots.

Shoddy claims of this sort could be avoided if journalists reporting on “fake news” attempted to follow reliable empirical methods in even a basic way — or at least the commands of basic rationality — by verifying whether the behavior that is claimed to be unusual actually is out of the ordinary. It is especially important to use valid methodology when reporting on allegedly “fake news” because misleading the public due to mistakes or exaggerated claims can lead to escalation of political tensions; ironically, deceitful attempts to identify and warn of the menace of Fake News, such as those peddled here by El Pais, can easily transform into a dissemination of Fake News.

(A full report with more detailed data regarding these empirical claims was recently submitted by one of the authors of this article, McGrath, to the U.K. Committee on Fake News.)

Reciting claims that match a commonly stated narrative, such as Russian interference in various western problems, is no substitute for fact-based analysis. One of the more methodologically unsound tactics used is to depict not only RT and Sputnik, but anyone who is quoted or even retweeted by them, as assisting in the spread of Russian state propaganda. During his testimony for the U.K. fake news committee, David Alandete from El Pais stated that “RT and Sputnik are at the center of this. Assange and Snowden are a very handy source for them; anything that Assange says is a quote and a headline.”

Assange was mentioned multiple times in RT and Sputnik’s coverage about Catalonia, but the stories quoting Assange comprised only a small minority of their discussions of these political events. Analysis of Sputnik and RT’s stories based on both Media Cloud’s data and their tweets reveal that only 1% to 3% of RT and Sputnik’s stories about Catalonia also mention Assange. Additionally, most of these references to Assange are centered around a few isolated quotes and events, in contrast to RT and Sputnik’s continuous coverage of the situation in Catalonia more generally.

Rather ironically (given the claims about bots and trolls promoting messages about independence in Catalonia), there is clear evidence of Twitter bots spreading messages about the crisis in Spain — but those were non-Russian bots and they were spreading propaganda that was opposed to Catalan independence. This is hardly the first time that Western governments and its allies have been caught using fake online activity to spread Western propaganda, but few western commentators care except when Russia or other U.S. adversaries do it.

Whatever else is true, it is likely that bot manipulation has been used in inflame the crisis in Spain — but they have been used to spread anti-Catalan messaging in line with Madrid authorities. On September 11, 2017, the user with the name @marilena_madrid tweeted a link to a story that Spain’s ABC published several months previously, which emphasized Puigdemont’s lack of legitimacy with EU institutions: a key anti-independence theme from Madrid authorities:

This @marilena_madrid tweet was retweeted over 15,000 times, but “liked” only 99 times. Researchers working on Twitter bot detection have discovered that bots often have low likes-to-tweets ratios of exactly this type:

In contrast to Assange’s tweets, which receive 1.14 “likes” per retweet on average — reflecting that they are spread overwhelmingly by actual humans — this anti-Catalan tweet from @marilena_madrid has a “likes” per retweet ratio of only 0.0062. A large number of the retweeters have random gibberish usernames such as @M9ycMppdvp5AhJb, @hdLrUNkGitXyghQ, and @fQq96ayN3rikTw, which also indicates that they may be bots.

Most of the accounts that retweeted @marilena_madrid, as well as @marilena_madrid’s account itself, now appear to be suspended by Twitter. Unfortunately, it is not possible to view data about suspended accounts except from on pages previously archived or cached, so there is limited data to study this particular set of bots in more detail. Thus, it is not possible to assess if this was an individual who had their tweets promoted by bots, a social media strategy by ABC to spread their stories with bots and trolls, or a state-sponsored propaganda campaign. Nevertheless, this case exemplifies the need for more multisided analysis about who is really using bot campaigns to propagandize the public.

It is certainly likely that one could find some actual Russian bots posting messages supportive of Catalan independence, but the magnitude has been wildly exaggerated by El Pais and many other Western institutions to the point of fabrication.

If, as appears to be true, these unsupported allegations about spreading disinformation during the referendum in Catalonia are being used as a tool for political manipulation in the case of Julian Assange, it is working. The escalation of tensions with Spain, which has strong diplomatic ties to Ecuador, threatens Assange’s asylum in a way that the longstanding pressure from the United States and United Kingdom could not. The intersection of these issues has lead to a rapidly deteriorating diplomatic situation, founded in part on exaggerated and inaccurate claims of disinformation during the referendum in Catalonia, where Ecuador is being forced to choose between maintaining their relations with other states and upholding Assange’s asylum.

The pattern of events seen here is not specific to Julian Assange, Ecuador, Spain, or any one country. It is a global, systemic problem. This situation an illustrative example of what happens when political tensions around internal divisions, in this case Spain and Catalonia, build and break. In the aftermath, people struggle to explain what they see as an injustice, and conclude that some foreign person or group interfered to bring about a problematic situation by spreading propaganda or disinformation.

This narrative grows and shifts the focus away from internal problems and divisions by unifying people against this new external enemy, much the way patriotism surges during a war. This sentiment can then be exploited as a tactic of manipulation for those seeking to support their own agendas, and leveraged to pressure external parties. It is also an extremely powerful tool for stigmatizing any internal, domestic dissent aligned with, if not controlled by, the foreign villain. Meanwhile, internal tensions continue to build and conflicts escalate, with their actual causes ignored in favor of pleasing, simplistic, self-vindicating storylines about foreign interference.

Correction: April 20, 2018
A previous version of this article incorrectly stated that the Spanish newspaper ABC was owned by Grupo PRISA. It is owned by Vocento.

Top photo: People gather to protest against the National Court’s decision to imprison civil society leaders without bail in Barcelona, Spain on Oct. 17, 2017.

The post How Shoddy Reporting and Anti-Russian Propaganda Coerced Ecuador to Silence Julian Assange appeared first on The Intercept.

AUTHOR.

(Photo: Battle of Nicopolis in 1396. Painting from 1523.)

http://JohnBatchelorShow.com/contact

http://JohnBatchelorShow.com/schedules

Twitter: @BatchelorShow

Before the Syrian crisis: The Fall of the Ottomans: 1 of 6: The Great War in the Middle East. by Eugene Rogan

https://www.amazon.com/Fall-Ottomans-Great-Middle-East/dp/0465097421/ref=sr11?s=books&ie=UTF8&qid=1524072317&sr=1-1&keywords=rogan+ottomans

"To have written a page-turner as well as an accurate and comprehensive history of the Ottoman struggle for survival is a remarkable achievement." --Wall Street Journal

By 1914 the powers of Europe were sliding inexorably toward war, and they pulled the Middle East along with them into one of the most destructive conflicts in human history. In The Fall of the Ottomans, award-winning historian Eugene Rogan brings the First World War and its immediate aftermath in the Middle East to vivid life, uncovering the often ignored story of the region's crucial role in the conflict. Unlike the static killing fields of the Western Front, the war in the Middle East was fast-moving and unpredictable, with the Turks inflicting decisive defeats on the Entente in Gallipoli, Mesopotamia, and Gaza before the tide of battle turned in the Allies' favor. The postwar settlement led to the partition of Ottoman lands, laying the groundwork for the ongoing conflicts that continue to plague the modern Arab world. A sweeping narrative of battles and political intrigue from Gallipoli to Arabia, The Fall of the Ottomans is essential reading for anyone seeking to understand the Great War and the making of the modern Middle East.

AUTHOR.

(Photo:Sultan Mehmed II's entry into Constantinople; painting by Fausto Zonaro (1854–1929) )

http://JohnBatchelorShow.com/contact

http://JohnBatchelorShow.com/schedules

Twitter: @BatchelorShow

Before the Syrian crisis: The Fall of the Ottomans: 1 of 6: The Great War in the Middle East. by Eugene Rogan

https://www.amazon.com/Fall-Ottomans-Great-Middle-East/dp/0465097421/ref=sr11?s=books&ie=UTF8&qid=1524072317&sr=1-1&keywords=rogan+ottomans

"To have written a page-turner as well as an accurate and comprehensive history of the Ottoman struggle for survival is a remarkable achievement." --Wall Street Journal

By 1914 the powers of Europe were sliding inexorably toward war, and they pulled the Middle East along with them into one of the most destructive conflicts in human history. In The Fall of the Ottomans, award-winning historian Eugene Rogan brings the First World War and its immediate aftermath in the Middle East to vivid life, uncovering the often ignored story of the region's crucial role in the conflict. Unlike the static killing fields of the Western Front, the war in the Middle East was fast-moving and unpredictable, with the Turks inflicting decisive defeats on the Entente in Gallipoli, Mesopotamia, and Gaza before the tide of battle turned in the Allies' favor. The postwar settlement led to the partition of Ottoman lands, laying the groundwork for the ongoing conflicts that continue to plague the modern Arab world. A sweeping narrative of battles and political intrigue from Gallipoli to Arabia, The Fall of the Ottomans is essential reading for anyone seeking to understand the Great War and the making of the modern Middle East.

The architect and urban designer Peter Calthorpe was an advocate of transit-oriented development (TOD) and smart growth long before those concepts were buzzwords. In fact, as one of the founders of the Congress for the New Urbanism and the author of the first TOD guidelines (as well as numerous influential books), Calthorpe has done as much as anyone to re-focus American urbanism on walkable, dense, sustainable, transit-rich environments.

Now, in an attempt to spread this knowledge even more widely, Calthorpe has released a new urban-planning software, UrbanFootprint, which will soon be available to virtually every city and government agency in California. (The company behind the software, which is co-led by Calthorpe and Joe DiStefano, is separate from the planning firm Calthorpe Associates.) Calthorpe compares UrbanFootprint to Sim City because it allows non-experts to model the impacts of different urban planning scenarios, such as zoning changes and road reconfigurations. He hopes it will help planners, politicians, and citizens’ groups communicate the benefits of the kind of urban environments he has spent his career advocating for and creating.

In the interview that follows, Calthorpe discusses some of the most pressing challenges in urbanism today, including NIMBYism, the emergence of autonomous vehicles, and the urbanization of the developing world.

What originally got you interested in urbanism?

I was born in London and I spent the first five years of my life amidst rubble and coal-laden air, kind of a nightmare domain. Then we moved to Florida and it was like the Wizard of Oz, you know—I landed, and the world became Technicolor.

We moved to Palo Alto in the early ‘60s. In those days, it was a landscape filled with orchards. As I grew up, I watched it erode into subdivisions and office parks, and it became clear that it was just being degraded. I guess I grew up being sensitive to the physical environment and the kinds of ways communities shape people and their sense of identity.

I became a radical teenager and I looked at what was going on from an environmental standpoint, and felt that there was really great harm at work in the way people were living. I just kept on from there. I fell in with Buckminster Fuller’s idea of whole systems: You just can’t think of one issue at a time.

I really quickly realized that urban design had a much bigger impact on all the important outcomes than individual buildings. So I started the Congress for the New Urbanism with Andrés Duany. What we proposed was a viable alternative to sprawl.

That gathered a huge amount of momentum, and as we made the case for the alternative to sprawl, we realized that we had to be able to speak in many languages. There was energy and carbon and climate change, or land or fiscal impacts to cities; mobility; air quality; health. The amazing thing about the city is that it touches everything.

The Bay Area when you were growing up was one of the most interesting places in the world: the explosion of the tech scene, the early hackers, the birth of the personal computer. And on top of that there was the counterculture movement and the music scene. How did those things influence you?

They totally influenced me. It was a community that connected the dots all the time. Stewart Brand was the guy who started the Whole Earth Catalog, but he also started The WELL [an acronym for “Whole Earth ‘Lectronic Link”], which was [one of] the very first email [services]. Stewart was a really good friend and mentor to me. The community of people who were willing to think in a broad way, but also to absolutely reject norms and posit completely different approaches, was ubiquitous. It was all over the place.

You had the Human Potential Movement in its heyday there. You had environmentalists who were really beginning to think about things as an ecology for the first time. And you had political activism, sometimes really extreme, like the Black Panthers.

The underlying principle was to question the norms, and when you question norms, you get innovation. That’s the foundation of innovation.

What do you think of what’s happening now in the Bay Area—its crisis of housing affordability and displacement, and the great shift in how the tech industry is perceived?

I don’t think technology and the city are at odds. I think that there’s a deeper phenomenon: We really still have not shaken the suburban sprawl paradigm that was born after World War II. The environmental community, some dimensions of it, are just anti-growth, which inhibits housing. Since the recession, jobs have grown in the Bay Area 11 times faster than housing. We’ve added over 600,000 jobs, and only 56,000 housing units.

But I also think that there’s something beyond that. It’s what we’re watching with Trump and all the rest of that tribalism, which is people being fearful and marking their territory and wanting to protect it. You know, a NIMBY is a little bit like a miniature version of Trump and his Mexico wall. It’s like he’s got a neighborhood and he wants to build a wall around it, and nobody else should be able to come now. We all have to realize this is a very powerful human emotion.

The only way you overcome that kind of feeling is with powerful coalitions. If enough groups can see common cause, and identify co-benefits and get behind a change, they can overcome that negative impulse to say, “Change is bad and I’ve got mine.”

You recently introduced a new urban planning software called UrbanFootprint. It’s got the backing of venture capitalists and is being used by the State of California. What makes it unique, and what kinds of projects will it be used for?

UrbanFootprint is a cloud-based software built to help planners, designers, architects, and advocates create sustainable, resilient communities. It supports a board range of stakeholders to enhance cities with the agility of data science and scenario-building. [They can use] UrbanFootprint’s extensive data library anywhere in the United States to assess existing environmental, social, and economic conditions in just a few minutes.

Once users get a sense of current conditions, they can lead community input to create alternative land-use and policy scenarios. Then they can evaluate their impacts across a range of key community metrics, including emissions, water use, energy use, land consumption, [pedestrian] and transit accessibility, and more.

We recently announced our partnership with the State of California, which will bring UrbanFootprint free of charge to over 500 cities, counties, and regional agencies. We’re thrilled to support California’s public planning efforts with this technology.

You’ve done a lot of work on transit and how it shapes urban form, and you have a growing interest in autonomous rapid transit. Tell us more about that.

The really exciting thing is that we’re going to have really affordable transit, in the form of autonomous rapid transit, coming online much quicker, quite frankly, than private autonomous vehicles. In dedicated rights of way, these things are safe, because they’re not mixing with ordinary vehicles. And because they’re driverless, the operation costs are so small that we can afford to build many more miles of this stuff.

We can basically grid our cities and suburbs with high-quality transit. The moment you do that, you have the armature for infill and redevelopment. So all the strict commercial environments—the six-lane roads lined with parking lots and single-story buildings, which, by the way, are losing value rapidly as Amazon steals all the economic activity from them—are these ribbons of opportunity throughout our metropolitan regions. By adding autonomous rapid transit and zoning for a range of housing types, we can solve the housing problem and the transportation problem.

But if you just let autonomous vehicles be privately owned or run as taxis, they’re going to make the situation much worse: anywhere from 30 percent to a doubling of vehicle miles traveled. It’s just like any amazing technology. If you use it well, you benefit, and if you use it poorly it can be catastrophic.

What do you think of this bill in California, SB 827, that would upzone all the areas near high-quality mass transit?

I don’t like the law as it’s written now, because it entitles redevelopment on single-family lots. I think that’s just a political formula for disaster. It won’t go anywhere. Disrupting existing single-family neighborhoods is really the last thing you want to do politically, and, quite frankly, [that you] need to do. The amount of land in ... single-story parking-lot environments, that nobody is in love with and that many people would actually like to see change, is more than enough to satisfy housing needs.

You’ve done a lot of work in China and the rapidly urbanizing parts of the world. What are some of the opportunities and challenges for good urbanism in these places?

In China, at the highest levels of government, the policy has completely shifted. Mixed-use, small blocks, [and] transit-oriented development are all now the norm and required. They appreciate that they can’t continue to add more ring roads. I think Beijing’s up to six of them now, and they’re all still in gridlock. And, by the way, in Beijing only 35 percent of households own cars. Just imagine if there was more auto ownership.

It’s just an impossible proposition, a high-density city whose mobility is dependent on private cars. It’s an oxymoron. It never works. And I think the Chinese really recognize that now.

Jane Jacobs, who had a very big influence on both of us, and who was so optimistic when she was young, called her last book Dark Age Ahead. Would you say you’re optimistic about the future of our cities in the United States and elsewhere? Or are you pessimistic?

I’m both. I think you can’t be intelligent and not be both. I am optimistic because cities offer the best, least-cost solution to so many of our challenges. Better than just technological fixes, healthy urban forms can resolve multiple issues simultaneously. For example, housing [units] placed in walkable mixed-use areas near transit and jobs cost no more than scattered subdivisions, but they are more affordable to homeowners, need less energy and water, cost less for cities to service, generate less carbon, and create stronger communities. These are the co-benefits I was talking about, co-benefits that can generate new, powerful coalitions.

I am pessimistic when I realize most cities don’t have the tools or processes to uncover these synergies—and too often default to piecemeal planning driven by professional and agency silos.

Democracy Now! 2018-04-13 Friday

• Headlines for April 13, 2018
• Syrian Researcher: Focus on Alleged Chemical Attack Ignores War's Ongoing Deaths by Airstrikes, Bullets
• As Trump Reconsiders TPP Stance, Fair Trade Advocates Say Real Fight Is over NAFTA Renegotiation
• Nearly 4 People Are Evicted Every Minute: New Project Tracks U.S. Eviction Epidemic & Effects

Download this show

See acast.com/privacy for privacy and opt-out information.

Today we chat with Dr. Keri Leigh Merritt about the topic of her new book, Masterless Men: Poor Whites and Slavery in the Antebellum South (Cambridge University Press, 2017).

Dr. Merritt is on Twitter as @KeriLeighMerrit and her professional website is https://kerileighmerritt.com.

You can join the discussion on the subreddit here.

Imagine you walk into a warehouse where the workers are on break, and you stumble into a vigorous, nuanced discussion of Marx’s notion of surplus value, how it relates to organizing on the shop floor, and how it applies to flexible and often female labor. Then the conversation turns to Gramsci and workers' councils in Turin. This is exactly what Carolina Bank Muñoz found when she visited a warehouse in Chile to study how unions responded to Walmart’s entry into Chile. The majority of Walmart's workers in Chile are unionized, and we talk to Carolina about her book, Building Power From Below, Chilean Workers Take on Walmart, and how Chilean retail and warehouse workers organized rank-and-file-led unions and win real economic gains along with respect and dignity on the job.

Then Nelson Lichtenstein joins the discussion on Walmart and organizing retail workers in the US. Nelson has several books on Walmart — as the face of capitalism in the twenty-first century, transforming American politics and business. Nelson anticipated a day of reckoning for Walmart as challenges to its "business model" grow at home and abroad — as the Chilean case shows. We'll also get Nelson's take on the "Amazon threat" and the state of US labor today in the Trump era.

Historian and political theorist Timothy Mitchell joins Dan for the first of a two-part interview on his book Carbon Democracy: Political Power in the Age of Oil, published in 2011 by Verso. In this first episode, we talk about how the rise of coal made both industrial capitalism and newly powerful worker resistance possible; and how the shift to oil then facilitated the persistence of imperialism in a decolonizing world while thwarting worker organizing. On the next show, we'll discuss a lot more, including how oil companies and Western governments made autocratic governments and conservative Islamists key partners in creating the very global order that we now find in such profound crisis. Thanks to Verso Books. Check out The End of Policing by Alex S. Vitale versobooks.com/books/2426-the-end-of-policing and Police: A Field Guide by David Correia and Tyler Wall versobooks.com/books/2530-police. And support this podcast with $ at patreon.com/TheDig, where you can also check out the first edition of our new weekly newsletter.

Democracy Now! 2018-04-09 Monday

• Headlines for April 09, 2018
• Glenn Greenwald on Syria: U.S. & Israel Revving Up War Machine Won't Help Suffering Syrian Civilians
• Glenn Greenwald: Brazil's Right Wing Jailed Ex-President Lula Because They Couldn't Win at the Polls
• "Apartheid, Rogue, Terrorist State": Glenn Greenwald on Israel's Murder of Gaza Protesters, Reporter

Download this show

AUTHOR.

(Photo: Map of the New York tribes before European arrival: Iroquoian tribes Algonquian tribes

http://JohnBatchelorShow.com/contact

http://JohnBatchelorShow.com/schedules

Twitter: @BatchelorShow

The First Frontier: 2 of 2: The Forgotten History of Struggle, Savagery, and Endurance in Early America by Scott Weidensaul

https://www.amazon.com/First-Frontier-Forgotten-Struggle-Endurance/dp/B00A123TK0/ref=mtaudiodownload?_encoding=UTF8&me=

Frontier: the word carries the inevitable scent of the West. But before Custer or Lewis and Clark, before the first Conestoga wagons rumbled across the Plains, it was the East that marked the frontier - the boundary between complex Native cultures and the first colonizing Europeans.Here is the older, wilder, darker history of a time when the land between the Atlantic and the Appalachians was contested ground - when radically different societies adopted and adapted the ways of the other, while struggling for control of what all considered to be their land.

The First Frontier traces two and a half centuries of history through poignant, mostly unheralded personal stories - like that of a Harvard-educated Indian caught up in seventeenth-century civil warfare, a mixed-blood interpreter trying to straddle his white and Native heritage, and a Puritan woman wielding a scalping knife whose bloody deeds still resonate uneasily today. It is the first book in years to paint a sweeping picture of the Eastern frontier, combining vivid storytelling with the latest research to bring to life modern America's tumultuous, uncertain beginnings.

AUTHOR.

(Photo: Iroquois painting of Tadodaho receiving two Mohawk chiefs

http://JohnBatchelorShow.com/contact

http://JohnBatchelorShow.com/schedules

Twitter: @BatchelorShow

The First Frontier: 1 of 2: The Forgotten History of Struggle, Savagery, and Endurance in Early America by Scott Weidensaul

https://www.amazon.com/First-Frontier-Forgotten-Struggle-Endurance/dp/B00A123TK0/ref=mtaudiodownload?_encoding=UTF8&me=

Frontier: the word carries the inevitable scent of the West. But before Custer or Lewis and Clark, before the first Conestoga wagons rumbled across the Plains, it was the East that marked the frontier - the boundary between complex Native cultures and the first colonizing Europeans.Here is the older, wilder, darker history of a time when the land between the Atlantic and the Appalachians was contested ground - when radically different societies adopted and adapted the ways of the other, while struggling for control of what all considered to be their land.

The First Frontier traces two and a half centuries of history through poignant, mostly unheralded personal stories - like that of a Harvard-educated Indian caught up in seventeenth-century civil warfare, a mixed-blood interpreter trying to straddle his white and Native heritage, and a Puritan woman wielding a scalping knife whose bloody deeds still resonate uneasily today. It is the first book in years to paint a sweeping picture of the Eastern frontier, combining vivid storytelling with the latest research to bring to life modern America's tumultuous, uncertain beginnings.

Historian and author John Julius Norwich reflects on some of the key moments in France’s history and relates a few of the more unusual and scandalous stories he uncovered while researching his latest book.

I realize it would be too much to ask that people who write on economics for major news outlets have any clue about how the economy works. I say that seriously; I have been commenting on economic reporting for more than two decades. Being a writer on economics is not like being a custodian or bus driver where you have to meet certain standards. The right family or friends can get you the job and there is virtually no risk of losing it as a result of inadequate performance.

But Roger Lowenstein performs a valuable service for us in the Washington Post this morning when he unambiguously equates the value of the stock market with the country’s economic well-being. It seems that Mr. Lowenstein is unhappy that Donald Trump’s recent tariff proposals sent the market plummeting. The piece is titled, “when the president tanks your stock portfolio.” It holds up Trump’s tariff plans as a uniquely irresponsible act because of its impact on stock prices.

Okay, let’s step back for a moment and ask what the stock market is supposed to be telling us. The stock market is not a measure of economic well-being even in principle. It is ostensibly a measure of the value of future corporate profits, nothing more.

Suppose the successful teacher strike in West Virginia spills over into strikes in other states, as now appears likely. Suppose this increased labor militancy spills over to the private sector and organized workers are able to gain back some of the money lost to capital in the last dozen years. That would not be good news for Mr. Lowenstein’s stock portfolio, but it would certainly be good news for the vast majority of the people in the country.

But this is the result of private actors, Lowenstein is upset about a president’s action’s tanking the stock market. Well, let’s give another one that would likely have an even larger negative impact on Mr. Lowenstein’s stock portfolio.

Suppose the next president announces that she will raise the corporate income tax rate back to 35 percent from its current 21 percent level. Any bets on what this does to stock prices?

Read More ...