Emacsen's Blog

Why OpenStreetMap is in Serious Trouble

2018-02-16T00:00:00-05:00

I was a contributor for OpenStreetMap for a long time, and I advocated for OpenStreetMap for a long time, but the project has stalled while the proprietary mapping world has continued to improve in data quality. For those of us who care about Free and Open data, this is a problem. In this article, I explore the reasons why I think OSM has stalled, as well as solutions to get the project back on track.

I was a contributor to the OpenStreetMap project from 2008 until roughly 2016. I was heavily invested in the project. In that time, I mapped, organized mapping groups in two cities, contributed to the founding of OpenStreetMap US, a non-profit dedicated to OpenStreetMap in the United States, gave talks on and about OpenStreetMap, contributed to the OpenStreetMap.org codebase, mentored two students for OSM through Google Summer of Code, started a working group dedicated to import of data into OpenStreetMap in the US, created and coordinated a massive bot run in the US, moderated the Reddit page r/OpenStreetMap and was a member of the OpenStreetMap Data Working Group, which gave me escalated privileges (both politically and technically) for the project. There are few parts of OpenStreetMap where I didn't have some either direct or indirect involvement.

I'm also the author of an article called Why the World Needs OpenStreetMap, which appeared in two large publications, The Guardian Online and Gizmodo, was on the front page of Hacker News twice and translated into at least four different languages. I was a proud OpenStreetMap advocate!

Before I criticize the project, I want to state emphatically that I still believe wholeheartedly in the core principles of OpenStreetMap. We need a Free as in Freedom geographic dataset just as much today as we did in the past. When I wrote my article about OSM in 2012, self-driving cars and other services were still a dream. Today the importance of having a highly accurate, libre geographic dataset is more important than ever, and I support those working to make it happen.

That said, while I still believe in the goals of OpenStreetMap, I feel the OpenStreetMap project is currently unable to fulfill that mission due to poor technical decisions, poor political decisions, and a general malaise in the project. I'm going to outline in this article what I think OpenStreetMap has gotten wrong. It's entirely possible that OSM will reform and address the impediments to its success- and I hope it does. We need a Free as in Freedom geographic dataset.

As long as this post is, it's not a comprehensive list of all the problems with the project, only the ones I found most directly affect the project's success and that I wasn't able to address myself during my time on the project.

When the World Needs a Map, Give them a Database

The first problem that I feel plagues OSM is that the OpenStreetMap Foundation views the mission of the project to provide the world a geographic database, but not geographic services. OSM gives people the tools to create their own map rather than offering them a simple, out of the box solution. Providing the ability for individuals and organizations to make their own map may work well for some, but it discourages small and medium size organizations from using OSM and thus engaging with the project. And even if they do use our data, their engagement is through a third party, rather than directly with us.

When you go to OpenStreetMap.org, you see a map and a few extras, such as a search window, along with a few extra buttons such as "Log In" and "Edit." It would be reasonable to assume that OpenStreetMap is a map, like Google Maps or other map projects, but while there is a map on OpenStreetMap.org, OpenStreetMap doesn't want you to use it. Instead, they want you to use the information from OpenStreetMap to make your own map, or find someone else to make the map for you.

If you find this strange or confusing, you're not alone.

A map is nothing more than a visualization of a collection of facts. We can understand this in terms of geometry. Let's imagine a furniture store called Frita's Furniture. Our map is a simple two-dimensional plane just like we had in geography class, and it's at 10,10. We might also imagine a road called Main Street that runs from location 2, 9 all the way down to 15, 9.

The location of the store and the road are geographic facts, but if we wanted to represent this data visually, we'd typically use a map. We'd choose just how to draw the road. Would we use a simple line or a more road-line picture? How wide should the line be? What color would it be? Where would we put the name of the road? We could put it across the line, or alongside it, or some other way entirely. And do we want to represent the store as a dot, or an icon of a store?

Years ago, map makers would handle this process manually, but with computers, we generally call this process map rendering, and there are many decisions around the rendering of a map, such the usage of the map, local conventions and even just keeping within a given organization's map style.

But most people don't care about any of this. They just want a map. OSM has a map on its website but discourages its use by third parties. Instead, users are expected to either find a commercial service to render the map for them or else do it themselves.

The project leaders claim that this is because they want people using OpenStreetMap to understand the difference between the geographic data and its visual representation and to encourage a free market ecosystem of rendered map providers, but it's also a fact that many of the individuals who push for this separation also sell commercial map services. I explore this conflict of interest later in this post.

Unclear Usage Policies

I mentioned earlier that OSM discouraged use of its maps on other websites. It does this through technically enforced usage policies. Understanding a usage policy is usually a straightforward process. An individual or organization gets permission to use a service a certain amount. We could imagine this being done by the number of map requests, or by bandwidth, etc. But OSM's usage policy is entirely different. They allow but discourage the use of the free map and then disallow any single application that is using over 5% of the map bandwidth. This policy is bizarre on several levels.

To understand why this is so strange, we can use an analogy. Let's imagine that I make ice cream. I put the recipe for my ice cream outside my house and suggest people make their own. I also offer free samples out of my home. Above my door, I hang a sign saying "Please don't ask for free samples". Then when people come in and ask for a sample, I give it to them. People may spread the word about my free ice cream and suggest their friends use it. Imagine we have a person named Fred who is a fan of my ice cream and recommends that all his friends go to my house for ice cream. I continue to dish out free ice cream to anyone who asks. But if one individual like Fred refers too many people to me, I will cut off access to everyone Fred sent.

Furthering this analogy, I will tell Fred's friends that they've eaten too much free ice cream, instead of telling Fred. And how many people is too many people? The answer for OpenStreetMap is anything over five percent of the total amount of free ice cream that I've dished out that day. Fred has no idea how many people I've served, so the only thing he can do is ultimately not refer people to my house.

This analogy works because no since no single service knows what any other service is doing, there's no way to know how many other applications have requested how many map requests. Also, since the top services and applications will change over time, you may be fine one day and in trouble the next. Again, there's no way to know. And when you do cross the line of using the service too much, your users would get an unfriendly message about unavailability, not you.

OSM could create standard usage policy, spelling out exactly how much free usage is allowed. It could also choose to create "premium membership" and encourage people to use its tile service (rendered map) service, but right now, using OSM tiles without going through a third party is hard.

A Bad Geocoder

When you type an address into a map and it gives you the location, that is called Geocoding. When your GPS or phone knows where you are and gives you a building or street address, that is called Reverse Geocoding. The geocoder featured on OpenStreetMap.org is called Nominatim, and it's awful.

Nominatim is not the only OSM geocoder. Much like the map rendering, it is possible to write your own or use a commercial geocoding service. But Nomatim is the most popular geocoder available for OpenStreetMap, it's used on the website and Nominatim is the service that is listed on OpenStreetMap.org under its APIs.

In its defense, let me say that Geocoding is hard and Nominatim itself is quite complex; its almost a feat of engineering. The developers who work on it put enormous effort into writing Nominatim. The problem is that such software needs to be maintained or sometimes replaced entirely to be useful. While there have been Nominatim maintainers, it's not been given the time or attention that it needs and deserves.

To understand why Nominatim is bad, one has to understand how most people use a geocoder. They're most often looking up a business or something vague like "Staples downtown Springfield". That simple three letter query is asking quite a bit. It is asking the computer to know what Springfield is and to limit the query to that. It's then asking to limit it to (or near!) an area called "downtown", and finally, it's limiting results to Staples.

But Nomintim can't handle such queries. It can barely handle simple address queries, such as "123 Main Street". As an example, if I typed an address into Nominatim near my location in New York City, it might come up with a result in Iowa, which for whatever reason, it's more inclined to offer me.

If I try to specify my location as "Manhattan", as of the time of this writing Nomintim will first assume that I mean Manhattan, Kansas, ignoring both the prominence of Manhattan in New York and the fact that the query itself is originating from New York City. Worse still, it's not possible to search for intersections. If I type "53rd and 6th, New York City" into Nominatim, it doesn't understand it. Even if I try to refine the search as "53rd Street and 6th Avenue, New York City", it doesn't work. Intersections are not addresses to Nominatim. It doesn't understand stores, or "near" or categories such as "restaurant". The results it comes up with are often irrelevant, and the service is quite slow.

While other geocoders for OSM exist, such as Pelias and Photon, only Nominatim is run and supported by the OSM Foundation.

No moderation/review model

One of the most significant technical problems with OSM is the lack of a review model, that is for a change to the map to be staged and then reviewed before being applied. Not having this functionality caused ripples of problems throughout the system, some of which I'll discuss here.

New mapper problem

Editing on OSM can be challenging for a beginner, and as the project tried to attract new mappers (editor contributors), we would run into people who just mapped incorrectly. Unfortunately, because OSM's data model doesn't include a review stage bad edits are committed to the map and often left undiscovered, or even if they're removed, the original editor doesn't usually see why.

Having the ability for a mapper to contribute changes and then have those changes be reviewed would have potentially left the map with higher quality data and a sort of mentorship model between new contributors and more experienced editors.

I hoped that this would be improved when I mentored a feature placed into OpenStreetMap called "Changeset Comments", in which users could leave feedback for one another's changes. Unfortunately, this ended up not being something many people used constructively, and it was a mess.

Without Moderation, Bots are Hard

Bots could be very useful in OSM in finding mistakes caused either by inaccurate data sources or by editing blunders. For example, if there was a road named "Main Street" connected to another road called "Main Stret," it was likely a spelling error and should be corrected. But it would be a good thing if changes were reviewed by a human being first.

But since OSM doesn't provide any mechanism for reviewed edits, these kinds of suggested changes don't exist. Either bot edits are executed without oversight, which could lead to errors, or they aren't done at all and the project misses out.

Imports are Difficult

Imports are challenging for a variety of reasons, but having a moderation or review model would make things much easier by allowing changes to be staged. The inability to stage and review massive changes to the map have caused problems in the past and many bad imports go unnoticed. If the project instead required that a human review edits before being committed to the system, bad imports could be detected before they cause problems.

Due to the lack of staging inside OSM itself, staging systems have been written for other OSM related projects. These systems often require editors to then manually place those changes in OSM. This process is labor intensive and makes some imports so challenging that they die before they begin.

Vandalism is hard to manage

Like Wikipedia, OSM has people who purposefully vandalize the project. Vandals have a variety of motivations. Some vandals are your run of the mill Internet trolls who enjoy causing problems. Sometimes a company wants something on the map despite community consensus against it and change the map to suit their needs, even if the project as a whole is against it. Sometimes mappers use OSM to make political statements such as in the case of disputed territories, and sometimes a geospatially based game, such as Pokemon Go, will use OSM to generate its data and players find that they can change the map to gain an advantage. Whatever the reason, OSM has vandals.

Vandalism is difficult in OSM because without a moderation system, it has to be "cleaned up" rather than prevented in the first place. Detecting vandalism is difficult. Several people ran monitoring tools to try to find problematic edits and imports. I was one of those people.

Even if problematic edits are detected, removing then means making even more edits. The history of the project is littered with lots of small changes that are only there to remove some previous change. Worse still, if the vandalism isn't detected early then someone else might modify an object that was previously vandalized, creating a situation in which either a tool or a person would have to separate the good edits from the bad, a manual process that can be labor intensive.

A moderation tool would prevent much of this. Many vandals would find that their work would not get into the database and would move on. While some malicious edits would still get through, we would be able to address a majority before it became a problem.

External Tools are Hard

One of my contributions to OpenStreetMap was working to improve MapRoutlette, a tool which helps find problems in OSM and offers users the opportunity to fix them. One feature that we wanted in MapRoutlette was to be able to present users with simple "Yes/No" type questions. Unfortunately, while not impossible this would have been a complicated task in OpenStreetMap. If these edits could have gone to a moderation queue, we could have been more confident, and possibly not needed MapRoutlette at all in some cases.

Many developers wanted to solve this same problem, offering the ability to add helpful but anonymous edits to the project. But since OpenStreetMap requires every edit be committed by an individual user, rather than a company or bot account, the barrier of entry for casual mappers was often too high.

OSM's Lack of Layers

Most geographic databases use a layered approach to represent different features. One layer may represent political boundaries; another may represent the road network, a third may represent water features, and so on.

Instead of the traditional layers, OSM chooses to use a single layer and then tags (key/value pairs) on individual objects. At first this seems like a good idea, but ultimately it ends up creating a huge mess.

Tools are Harder to Write

Imagine if we wanted to write an editor for OSM that only worked with the road network. This task would seem straightforward in that we would just need to extract features that correspond are tagged as road, such as highway=*. Unfortunately, it's not that simple.

First, an editor that edits these road features must not only pick up the roads (ways in OSM terminology) but also the points (nodes) that make up that road. Secondly, if the road is particularly complicated, it may be represented as a relation. Editing this way is time consuming but straightforward.

What is not as straightforward is that a feature such as a road may also be playing double-duty as another feature, such as a political boundary, as may any of its associated features. Editing roads may inadvertently result in changing a political boundary.

Having map features that represent such radically different meaning puts an onus on both tool makers and the individual editor working on OSM to be aware of any changes they make possibly having consequences that go beyond what they think they're doing.

Imports Are Difficult Without Layers

One of the keys of the Free and Open Source software movements have been code reuse, the idea that you can integrate software together from different sources and have it work seamlessly together. One would think that it would be much the same with geographic data, but because of the lack of layers, it's very challenging to import data into OSM.

Without layers, it's difficult to extract a specific region by feature and analyze or replace that. Instead, because of its complex tagging system, it needs to be analyzed as a whole. Imports are possible but made more difficult without layers to make the job of data analysis by isolation easier.

No Support For Observational, or Other Datasets

One of the core tenants of OpenStreetMap is that it only stores persistent, personally verifiable data. The only exceptions to this are political boundaries- and even these exceptions can be problematic. Unfortunately, this also presents a problem with third parties want to use OSM for things outside of the project scope.

As an example, let's take Pokemon Go. Pokemon Go is an augmented reality game in which real-life features are connected with imaginary creatures in which the player must battle and collect. The frequency and location of where these creatures appear is based on various map features.

Pokemon Go players wanted to use OSM to document the location of creatures to make it easier for other players to find rare creatures and improve their collection. OSM disallows this kind of data in the same way that it might for bird watchers- while it's interesting, the impermanence of this data made it a poor candidate for the project and thus the data would be immediately removed.

But it doesn't have to be something as trivial as a game- layers could also allow other specialized data such as potholes, red light cameras or even bird or animal sightings. It would make the project useful to many more people.

Lack of Permanent IDs

In any database, objects have an ID field, usually a numeric value to look the record up by. OSM is no different and every object inside OSM has an ID field. Unfortunately, in OSM the ID fields represent the low-level objects rather than any high-level concept. This creates a huge problem. I will call this idea a "Conceptual Object", and show how the lack of permanent IDs for them is problematic.

Diving Deeper

To understand why the lack of permanent IDs is a problem, we have to dive a little deeper into how OSM works. While many of these low-level details are beyond the scope of this article, I will present the basics of how OSM stores information. A point in OSM is called a node, and every node has an ID. Points may be collected into a line, and that line is called a way, and collections of nodes and ways may be combined into a more complex object called a relation. A relation may also contain other relations. Nodes, ways, and relations all have ID fields.

To illustrate this, let's think of a building, that building has properties. The building is at a certain location, it's a specific size, shape, and has an address. If it's large enough, it may have multiple addresses. But the concept of the building is unified. In OSM, that building could be represented by a single node, representing the address. Or a building may be represented by a way of the building outline, or a building may be represented by a relation, encompassing details of the various building elevations, levels, and roof types. The problem is that if I'm doing a lookup, there is no straightforward way to ask about the building. Instead, I will have to look at aspects of the building, such as its address, or its location.

History Lost

As strange as this may seem, it's entirely possible in OpenStreetMap to take a node from one side of the world, move it to the other side of the world and use it for something else entirely. For example, it is technically possible to take a part of a house, move it to another continent and use it as part of a road. While this is highly unusual, this is not disallowed. If I look at the history of the node, I will see it move. While this may keep the history of the element, it does not keep the conceptual history of an object.

For example, if we start with a representation of an object being the building as a single node, then move to a complex relation, that won't be reflected in the object history, and thus the changes over time are lost.

Permanent IDs on conceptual objects could help with this by providing a history of what the data represents rather than just the data itself.

Import Conflation

Amongst other problems, not having a permanent ID for a conceptual object in OSM is the challenging of conflating objects in OSM to objects in other datasets. For example, if we're given a building database from a local government, each building in that dataset will have an ID. We will want to compare that ID to our existing objects. Unfortunately, to do that, we're left with two choices- either we create a new identifier (key) in which to do the conflation, or we have to use the second dataset's ID inside OSM- neither of which is an optimal solution.

It's Hard to Build Connections to Other Datasets

Many people have envisioned projects that connect to OSM to offer reviews or other data associated with OSM, but without a permanent ID, this is not practical. Objects in OpenStreetMap may contain some data such as cuisine type or opening hours along with the name and address, but the review site will need to be able to have a permanent link to objects in OSM, which it can't currently.

No Standards in Data Representation

In OpenStreetMap, there are no formal standards in the project for the representation of features on the map. As an example, let's take the example of a sidewalk. Sidewalks are useful things to have on a map because they tell us if the road is pedestrian friendly. Sometimes sidewalks are represented by an attribute on the road itself. Sometimes sidewalks are represented as a line (way) that runs parallel to the road. Sometimes those ways have the name of the street as their own name, and sometimes they don't have any name at all.

If you are a mapper, this is confusing, since there's no one standard way to map things. If you're trying to build tools to work with OpenStreetMap, the lack of standardization of data across the project makes it challenging to work with as a whole.

There is an informal process for data representation, mainly done on the Wiki, but because this isn't formally enforced, and changing data en mass may be considered a form of vandalism, data consumers are forced to write tools that accept many representations of the same data.

The APIs are Slow to Evolve

As of writing, the current official OpenStreetMap API is 0.6. The API hasn't version hasn't changed since 2009. While a stable API can be a good thing for a mature software project, in the case of OpenStreetMap, this is as more a reflection of poor project management.

An API is part of a protocol that either allows a client to talk to a server or for servers to communicate with each other. In this case, we're talking about OpenStreetMap's editing API which is used between OSM and editing software.

The OpenStreetMap editing API is very powerful and complete, but it has some design choices that made sense in 2009 that have largely been replaced by better technical options in nine years since. These include small changes, such as the data serialization format, as well as more significant changes such as the internal data representation.

As an example, in 2012 there have been several proposals made to create a new datatype called an a area that would greatly simplify the representation of certain types of geographic features. Despite this and the offer of technical help, the project has not made any significant progress on this or other important technical issues.

OSM has Hidden Gatekeepers

Dovetailing on the previous section, we have to ask why the project has not made more technical progress, and the answer is that sadly the keys to the OSM castle largely do not lie in the hands of the OpenStreetMap Foundation, but instead in the hands of one or two individuals who act as gatekeepers to the project's source code and infrastructure.

While it's not uncommon for a Free Software or Open Source project to have a "Benevolent Dictator for Life", these roles are often replaced by a more formal structure as the needs of the project grow. In the case of OpenStreetMap, there is a formal entity which owns the data, called the OpenStreetMap Foundation. But at the same time, the ultimate choices for the website, the geographic database and the infrastructure are not under the direct control of the Foundation, but instead rest largely on one individual, who (while personally friendly) ranges from skeptical to openly hostile to change.

As a former professional system administrator, I relate strongly to these types of individuals. At the same time, the desires of them need to be balanced by the overall needs of the project to make progress and keep momentum to keep its userbase happy and engaged.

That is not the case here, and it's to the detriment of the project.

The OpenStreetMap Foundation Culture

It would be easy to think about the OpenStreetMap Foundation (the OSMF) as similar to the Wikipedia Foundation, but aside from the high-level view of being the holder of Free Data, the two projects are managed radically differently.

The Wikipedia Foundation is a multi-million dollar organization that not only manages Wikipedia but other projects as well, such as the lesser known Wikidata and Wikinews. These projects aid in the organization's broad mission to provide high-quality information to the world. To serve this mission, Wikipedia spends a great deal of money on its infrastructure as well as directing and funding development of new tools for the community to use.

OpenStreetMap, on the other hand, relies primarily on donated hosting services and runs on a shoestring budget. It has no paid employees and does not fund or direct the development of its software base.

This has lead to some organizations trying to take up the mantle and improve the situation, including an organization that I helped found called OpenStreetMap US, which is a US based non-profit organization focused on promoting OSM in the United States. Among our goals for the organization was to fill in the gaps of development and mapping resources by the OSMF, which we partially succeeded in doing, but because of the fragmentation of organizations, we were less successful than we hoped.

In addition to OpenStreetMap US and other "chapters" around the world, there is the Humanitarian OpenStreetMap Team, whose mission it is to help promote OSM in developing nations and rally the OSM community during humanitarian crises. There is no reason that HOT needed to be an independent organization other than the unwillingness by the OSMF to expand its role. Even Steve Coast, one of the founders of OpenStreetMap saw and tried to address this problem with his organization, "Map Club."

The obvious question is why the OpenStreetMap leadership takes the positions that it does, despite the clear need for change. The answers in my view are commercialism in the project, along with a cultural desire to retain the feel of the project's early days.

While there are companies built around Wikipedia's engine (the Wikimedia Server), there are not many companies making money from repackaging Wikipedia. OpenStreetMap, on the other hand, has a commercial ecosystem around it, largely from the business of creating customized maps for customers.

Many of the founders of the project, as well as others, have launched commercial services around OSM. Unfortunately, this creates an incentive to keep the project small and limited in scope to map up the gap with commercial services which they can sell. This also applies to HOT, which has a financial incentive to get grant money for itself and not have those resources going to the OSMF.

In addition to these conflicts of interest is a desire to keep the project small in scope by senior members of the community who see the project as being about people and the mapping hobby and want to avoid imports or other activity that could be seen as removing the human factor from the project. They also see the dangers inherent in creating an organizational structure that demands money and fears it would create a perpetual cycle of needing to find donors simply to support a management layer.

I disagree and view the lack of a more active structure by the OSMF as the cause of the project's both stagnation and significant commercial influence.

The World Had Changed

When OSM was launched, governments did not release their data under free licenses. They only began doing so because OSM exists now as competition. Yet due to the problems I've outlined, OSM imports are difficult, and updating imports, once they get in OSM is nearly impossible. This is a critical problem for the project.

Similarly, when OSM was launched, drones were not cheap and available. AI wasn't able to do good visual detection of roads, and flying cars were still science fiction. Now all of these tools exist, and yet OSM is still stuck largely editing by hand.

If OSM relies exclusively on manual labor and be unable to work with other datasets, its data quality will continue to decline and the project will ultimately stagnate and fail.

Just the Roadblocks

It may appear at first that this article is a comprehensive list of everything I find wrong with OSM. It's not. There are many more concerns I have about the project, but I've limited my article to the scope of concerns I have that I feel are stopping the entire project from progressing. There will be time to fix the small issues if (and only if) the project as a whole succeeds. If it doesn't, then the small nit-picky problems are going to be irrelevant anyway.

It's my sincere hope that this article will be a call-to-action for OSM. There are many brilliant and inspiring individuals in the project. If I'm am a pun, I hope OSM will once again find its way.

Free Software Call To Action

2016-11-27T11:28:00-05:00

This last year has been a very difficult one for civil liberties and personal freedoms.

Donald Trump, the president-elect of the United States, has promised to round up and deport illegal immigrants as well as removing legal immigrants from the US. In addition, and more importantly, he's discussed creating a "Muslim Registry" and his cabinet has likened to the US Japanese Internment Camps of WW2, and supports mass surveillance. Pence, the vice-presidential nominee, supports rolling back protections for gays, lesbians and transgender people.

With these views espoused by the political leadership, it should be no surprise that hate crimes are on the rise, with the Southern Povery Law Center documenting over 700 hate crimes in just the week after the election.

Brexit symbolized a change for the United Kingdom, signaling a change from inclusion and connection with Europe to isolationism. Racist and xenophobic statements were part of the campaign to disconnect the UK from the European Union, drawing on ideas that in the past, the United Kingdom was strong, and could be again. In the last month, the UK has passed a law that gives the government not only the ability to track its citizens, but access to all their past activities online, including browser history, emails and other communications.

Aside from the US, right wing parties are on the rise in Europe, with the rising popularity of Frauke Petry's AfD party in Germany, Norbert Hofer's Austria Freedom party in Austria as well as Marie Le Pen's National Front in France. All of these political movements signal a move away from inclusion and social acceptance and towards isolationism, "traditional values" and the persecution of vulnerable populations such as refugees, immigrants, gays and lesbians, transgender people, people of color, Jews, Muslims, the physically handicapped or the neuro-divergent. If you aren't personally mentioned in this group, it's likely that you know someone who is- and believe me, they, I, am scared.

We're scared that our lives will be destroyed, that the work we've done to make our lives better will be undone, that marriages will be dissolved, that employment protections will be rolled back and we'll be fired, or worse, that we'll be forcibly taken from our homes, physically attacked or killed.

With these established facts, what practical steps do we take from here? I outline four practical steps that we Free Software people need to take.

Establish and promote alternative communication platforms

The first step the Free Software community needs to take is to recognize for ourselves that we need free methods of communication. Freedom is not free unless we can communicate with one another, whether that be messaging, audio/visual communication, or the distribution of static content.

Because of the enormity of this task, I've broken it down further into smaller sub-tasks.

A. Promote "small step" improvements in communication security.

We in the Free Software community have a tendency towards absolutism. There are understandable and justifiable reasons for that, but when the stakes are so high, we must not become so entrenched in [TK!] that we lose the opportunity to help others through smaller steps that they could take right now, without making a major change in their lives.

Asking all your friends to use OpenPGP and plain text communication might be asking too much, but sending people to Signal and encouraging them to use it is a good first step. Similarly, encouraging the use of technologies like TOR, and pointing them to the EFF's Secure Messaging Scorecard are great ways to make small improvements, which is our first step in making the kinds of progress that we need in ensuring our rights to communicate and congregate freely online.

B. Test, use and promote more comprehensive communication improvements

When a new communication platform launches and it respects your privacy, use it, and encourage others to use it. It may be difficult or impossible to get people to add something onto their existing communication platform, but with new platforms come "new rules" and it may be easier to encourage them to switch to a new system than retrofit their old one. You can help test these platforms and encourage their use by example.

C. Use and Promote Self-Hosted Communication Platforms

A big problem right now is the use of third party hosted platforms for our communication. Google and Facebook and others are very convenient, but result in a loss of privacy and control. Instead, self-hosted platforms (even on a commercial platform such as a VPS) are better as they decentralize your data.

If you are in a decision making capacity at your job, neighborhood association or other organization, encourage and support the use of self-hosted tools.

If you're a developer, consider the ease of use of deploying and maintaining the service for very small groups of 2-20 people. Use defaults that work well for small groups and consider small libraries such as SQLite instead of requiring a standalone database. If you can, provide a containerized version of your application for easy installation.

If you're a system administrator, consider helping provide helpful tutorials and scripts to make the deployment of software easier.

D. Offer support to alternative platforms

In the 90s, the community had social gatherings called Installfests where people came together to offer technical support and assistence in helping bring new people to Free Software. If we have entirely new operating systems or hardware, we may need to bring these Installfests back, not just for their role in technical assistance but also as an opportunity to make one on one, personal contact and build a local community. It's important for people to feel connected and part of something larger than themselves, and offering physical meetings can play an important role in realizing that goal.

Recognize and support community diversity

More than product developers, the Free Software community is built on top of a shared experience and shared culture. Part of that culture is acceptance and welcoming of people who are different from you. We need to stand by our community members, no matter where they're from, their religious background, their gender identity or sexual orientation. We are stronger together.

While these are nice sentiments, they must also be backed up with action, especially in regards to our most vulnerable populations. In some cases, that may mean codifying these principles into a community statement or code of conduct, but in all cases, it means that if you see something wrong, you need to stand up to it. We need to ensure that the feeling of camaraderie, familiarity and safety that most of us feel within the community is something shared with everyone.

Communicate openly and loudly about the moral, legal and ethical threats to our liberties

Free Software has always been about more than code, it's been part of a larger moral and ethical framework for understanding the world. We understand the role of Free Software in a larger [TK!] of personal liberties and freedoms. Free Software is a requirement for free communication in the online world. We need to know that our devices are doing what we want them to (and not what someone else might want them to do do). Once dismissed as paranoia, we know for a fact that modern "cloud enabled" devices transmit personal details about us, including our location, our interests and even record us.

If we value our freedom, we need to be able to establish boundaries around the encroachment on our civil liberties. In this time, when encroachment of civil liberties has become a norm, it is even more important to demonstrate the importance of this issue to other groups, and to build these principles into their mission statements and guidelines. To be clear, this is not about "Run Linux everywhere!", this is about codifying the principles of freedom and liberty in the framework of other organizations and by demonstrating the important of Free Software in carrying out that mission.

Get involved in politics

National politics is difficult and often inaccessible, but local politics is often easy to get involved in. Even small towns have many elected positions, and small changes even on a town or city level can make a difference, if only by bringing awareness to an issue. We must get involved in out political process, not by trying to change the minds of politicians from the outside but by being involved in the political process from the inside and using our influence.

This can be at every level. There are many articles online about getting involved in local politics, so I won't replicate their efforts here.

We must take a comprehensive approach to taking back our society. We owe it to ourselves and others to get involved not only to make our voices heard, but to protect ourselves and our loved ones.

We, the Free Software community, are a community of shared values and goals and now, more than ever, we need to turn those ideals into action.

Why is SFLC siding with Oracle over Linux developers?

2016-02-28T20:36:38-05:00

I don't write about politics much on this blog (in fact I haven't written anything in quite a while), but Free Software is something I care a great deal about, and so when I see something fishy and scandalous, I take interest, and that's exactly what's going on with the Software Freedom Law Center's position on ZFS's license compatibility with Linux.

If you haven't heard, the Software Freedom Conservancy made a public statement about ZFS. The facts can get a little complicated but in a nutshell, Cannonical, the company that makes Ubuntu, is going to distribute ZFS with the next release of their operating system (16.04). Unfortunately, ZFS (which is owned by Oracle) is distributed as a separate module under a licensed called the CDDL, which is non-GPL compatible. You can read Conservancy's post here: https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/

There's a simple solution to this problem. Since Cannonical can't ship this code legally without infringing on the Linux (and possibly ZFS) license, Oracle should simply change the licensing of ZFS from CDDL to another, GPL-compatible license, either the GPL2 or a permissive license such as the MIT license. It would be a relatively correction for Oracle to make and it would allow the distribution of ZFS.

What's more interesting than this conflict is the response that the Software Freedom Law Center posted an article on thier website about the case: https://softwarefreedom.org/resources/2016/linux-kernel-cddl.html

I imagine that if you've read it, you're probably thinking that's it's incredibly well written, but you may feel vaguely dizzy. I understand; I feel that way too when I read Eben Mogen's writing. His approach to language can be incredibly inspiring, but at the same time by using so many words, he creates a cloud of obfuscation of the facts. Usually I'd say that I'm glad he's on our side, but this post of his is decidedly not on our side.

For all its length, the article has two main points. Firstly, it says that the CDDL (the licensed used by Oracle) is not compatible with the GPL in fact, but it is so "in spirit". Secondly, it says that the kernel developers are the problem, not Oracle.

To the first point, that the CDDL is close in spirit but not in fact is an odd one for a lawyer to make. As former councel for the FSF, Moglen has an understanding of the terms of the license that most people don't have, and he knows very well that once the license is committed to writing, we use what's written to decide whether or not something is allowed under the license, rather than the ambigious idea of "the spirit of the license", which he refers to in his post.

Which gets to Eben's second point, which is to shift the blame of the problem onto the kernel developers, rather than onto Oracle. What he's saying, in essence, is that he would like the GPL to be different than it is. This reminds me of those people who argue that the GPL is a de-facto public domain license, due to its spirit. It's not. The GPL spells out its terms very clearly, and Oracle certainly has lawyers that understand the GPL.

Which leads me to my third and final point. Eben Moglen is a genuis. He's someone that I admire immensely and have at time been inspired by and aspired to his skills as both a thinker and an orator. Knowing his history and feeling as I do about him, this post seems entirely out of character. So then if it's out of character, why would he do it? Why would he side with a large, powerful corporation over GPL-using kernel developers? The only conclusion that I can come to is that he must be recieving money from Oracle. I'd love to be able to either support or negate this but the Software Freedom Law Center does not publish the names of the companies that donate to it. In other words, we can't know what corporate interests have donated to SFLC because SFLC had decided to not disclose its corporate donors.

The Rule of One

2015-03-31T13:44:10-04:00

If you care about Free and Open Source Software, you probably think about ways to support the people who write the programs that you rely on. In this post, I'm going to tell you what I do, and what I hope others will do too. I call it the The Rule of One

The Rule of One is very simple:

Whatever you spend on proprietary software, donate that same amount to Free Software causes.

If you're well off, you should employ the Rule of Two to One

Whatever you spend on proprietary software, donate double to Free Software causes.

Think about all the times that we buy proprietary software:

Did you buy a game for your smart phone recently? Or for that matter, did you buy a smart phone that came with proprietary software on it?
Did you buy some proprietary hardware, like a Roku , or maybe a "Smart" television? Or even a medical device?
Did you buy a video game or game console system?

If you consider all the times that we either purchase or subsidize proprietary software, for many of us, it ends up being hundreds, possibly thousands of dollars a year.

I'm not telling you to stop buying the things you love. Instead, I'm saying that whatever you're spending on proprietary software, invest in creating the Free alternatives.

If you're not sure how, most of the larger projects have a Donate button on their home page. Alternatively, here are some organizations that support Free Software that you should consider donating to:

Modern Web Development with React (Part 4)

2015-02-08T12:57:13-05:00

In the previous post, we created our first React component. In this post, we'll build on that and explore the issue of state in React, embedding components into components as well as React's component lifecycle.

Immutable Properties

When we left off, we had explored React's concept of props. Props are constants that we can send to a react component at its initialization time and they remain with the component.

Properties do not change. If we want to create a component with a new property, we would create a new component.

Default Properties

Getting back to our Temperature component, we have to manually set our temperature property. If we we to fail to set that property, our web page would break, possibly not showing anything at all, because the props.fahrenheit value would be undefined.

We can set a default property on our component in case a property isn't sent to it.

For that, we'll create a getDefaultProps method, which will return a Javascript object containing default properties. In our example, we'll set a default temperature of 72F.

To do that, create a new method for our Temperature component:

getDefaultProps: function(){
    return {
        "fahrenheit": 72
    }
},

Now we can either set the temperature explicitly as we did before, or we can omit the temperature property and a default value will be set for us.

Getting dynamic with state

So far we've been using a static temperature that we've either been setting manually or using a default value for. Now we're going to create a component that is more dynamic.

Let's imagine that we have an RESTful API /temperature which returns a JSON response in the form of {"fahrenheit": 80}. Instead of setting our component manually, we could use this API endpoint to set our temperature.

While we could use props for this, if we only intend to set the temperature once, we're going to go ahead and use state instead.

React components' state property works similarly to the props property except that values in state are changeable using the setState method, which we'll use. It's important to always use setState to set the values inside state, rather than trying to set them directly.

We'll want to fetch the current temperature as soon as we can. The best time to do this is right before the component will mount to the DOM. So we'll set the React method componentWillMount to retrieve our value and set the temperature. We'll also change our render method to use the state, instead of the prop.

var Temperature = React.createClass({
    componentWillMount: function(){
        request.get('/temperature', function(res){
            this.setState({fahrenheit, res.body.fahrenheit})
        }.bind(this));
    },
    render: function(){
        return (
        <h1>The temperature is {this.state.fahrenheit}F<h1>
        )
});

What's Happening Here

Although we made a small change, there's quite a bit to go over.

Out componentWillMount method will fire just before the component is rendered in the browser, and it will send off a RESTful call to the URL /temperature.

Once that's complete, it will fire off the anonymous function that we've passed request.get, which calls the this.setState method with the newtemperature property.

this returns

You may notice that we've added .bind(this)to our anonymous function call to request.get. That's because we call the this.setState method on it, and this is always set to the current function namespace. We can get around that by binding this, our component, to the anonymous function- thereby giving it access to all the properties of our component, including the setState method.

Look Ma, no render()

The next thing you may notice is that we didn't explicitly call render() after we set our new state. That's because React will do that for us. Once we set a new state, the render method will get re-run and, if necessary, the page will be refreshed. More on this later in the post.

If you've had to work directly with DOM manipulation, then you'll see the clear difference here. We only need to worry about the state of our component through properties and state, and the component will take care of its own display.

There's still a problem

If you run this code, the render() method will be executed while the request.get is still waiting. That's the asynchronous nature of Javascript, and it's generally a good thing.

The problem for us is that in the meantime, our render method will return something like:

The temperature is undefined, or worse, may fail to run at all.

There are two ways we can solve this. The first is that we can set a default temperature, and the second is that we can put some logic in our render method to handle this condition. Let's go through both.

Setting Default State

To start, let's just set a default temperature just like we did in the previous post. For that, we'll use the getInitialState method.

Add the following method to our component:

getInitialState: function(){
    return {
        fahrenheit: 72
    }
},

When the component runs, for a brief period, it will display the temperature as 72 until the request call completes.

Using logic during render

Instead of setting a fake temperature value, we can add some logic to our render method telling the user that we're fetching the temperature.

Let's do that using the original component as displayed early on in the post:

var Temperature = React.createClass({
    componentWillMount: function(){
        request.get('/temperature', function(res){
            this.setState({fahrenheit, res.body.fahrenheit})
        }.bind(this));
    },
    render: function(){
        if (this.state.fahrenheit === undefined){
            return (<h1>Getting temperature...</h1>)
        } else {
            return (
            <h1>The temperature is {this.state.fahrenheit}F<h1>
           )
       }
});

Now while the request is in process, the message Getting temperature will be displayed to the user.

Combining default state with render logic

It's nice to be able to set a message that the temperature isn't yet set, but using undefined is probably not a good idea. Instead, let's set the state of fahrenheit to null. Because null is distinct from undefined, we can test for it explicitly.

So let's do that:

var Temperature = React.createClass({
    getInitialState: function(){
    return {
        fahrenheit: null
        }
    },
    componentWillMount: function(){
        request.get('/temperature', function(res){
            this.setState({fahrenheit, res.body.fahrenheit})
        }.bind(this));
    },
    render: function(){
        if (this.state.fahrenheit === null){
            return (<h1>Getting temperature...</h1>)
        } else {
            return (
            <h1>The temperature is {this.state.fahrenheit}F<h1>
           )
       }
});

Flexibility

We're using the url /temperature for our RESTful call, but that URL could be anything. In fact, we could be setting the state from any function call at all. That's in contrast to some other libraries which dictate exactly how the server needs to lay out its data. Here, we have the ultimate control over that and we can choose an API that makes sense for us.

Multiple components

So far we've only created a single component, but we can create multiple components.

We're going to create one component that displays the temperature in Fahrenheit and one that displays it in Celsius.

var Fahrenheit = React.createClass({
    render: function(){
        return (
        <h1>{this.props.fahrenheit}F</h1>
        )
    }
});

var Celsius = React.createClass({
    render: function(){
        var celsius =  (this.props.fahrenheit - 32) * 5 / 9;
        return (
            <h1>{celsius.toPrecision(1)}C</h1>`It
        )
    }
});

var Temperature = React.createClass({
    componentWillMount: function(){
        request.get('/temperature', function(res){
            this.setState({fahrenheit, res.body.fahrenheit})
        }.bind(this));
    },
    render: function(){
        return (
            <div>
                <Fahrenheit fahrenheit={this.state.fahrenheit} />
                <Celsius fahrenheit={this.state.fahrenheit} />
            </div>
        )
    }
});

Reusable Components

Instead of creating the rendered output directly in the Temperature Component, we now use the Fahrenheit and Celsius components. Using JSX we can treat these components just like regular HTML in our application. This level of abstraction is very useful as we build larger and more complex applications.

In addition to the ease of creation, we do not need to worry about our children object's state. That is handled for us in our Render method. That means that at each component level, we can focus our attention exclusively on that component without worrying about what either the parent component or the child component are doing.

Using Props

You may notice that we're using props for our temperature display, rather than state.

That's because we know that the temperature for each of these components is not going to change, so props make sense. The props for the components does not change, instead, new components are generated.

Making our code more clear

Having the data retrieval in componentWillMount works, but it's not entirely clear initially what's going on. Let's make it more clear by adding a new method to Temperature that explicitly says that we're fetching the temperature:

getTemperature: function(){
    request.get('/temperature', function(res){
        this.setState({fahrenheit, res.body.fahrenheit})
    }.bind(this));
},

Now we'll reference this method in our componentWillMount, so it will become:

componentWillMount: function(){
    this.getTemperature()
},

Custom Methods

Writing our component this way has an added benefit of more readability. It's more obvious what's going on now inside of componentWillMount.

Because React components are normal Javascript objects, we can add new methods to them. The React framework gives us guidelines but generally stays out of the way.

Getting Updates to our Temperature

What if we wanted to get regular updates to the temperature? We can use Javascript builtin setInterval to regularly retrieve the temperature from our API endpoint and update the temperature accordingly. Now that we have getTemperature as its own function, we can do that pretty easily.

We'll change our componentWillMount method to add the setInterval call. setInterval returns an identifier to keep track of the timer- so we'll store that timer in a state variable in case we want to use it later.

componentWillMount: function(){
    var intervalID = window.setInterval(this.getTemperature, 30000);
    this.setState({intervalID: intervalID});
}

So now the component will re-load the temperature from the server every 30 seconds.

The Shadow DOM

You might think that destroying an old component and creating a new one would be slow, but React optimizes this by creating a copy of the DOM in memory and then only editing that DOM. It then create a diff of the two and only modifies on screen what needs to be changed. This is done incredibly quickly, so screen refresh times are very short, in the tens of milliseconds.

Cleaning up after ourselves

In our case, we do not destroy the Temperature component without also closing the window, but it's possible for us to do so in another setting. If we were to do that, the browser would still make the getTemperature call every 30 seconds, because that function is bound to the window object, not just the Temperature instance.

To address this, let's add a new method to our Temperature component that removes the interval call upon component destruction. The method name for this in react is componentWillUnmount:

componentWillUnmount: function(){
    window.clearInterval(this.state.intervalID);
},

Now we're free to create and destroy as many Temperature component instances as we like.

Summing it up so far

In this post we covered using React state, we covered making components that have children components, integrating React components with external data, as well as using the React lifecycle methods to create methods that execute before and after the component mounts.

In the next post, we'll go through how React makes forms easier, as well as touch on using CSS frameworks to help us build our applications more easily.

Modern Web Development With React (Part 3)

2015-02-06T18:12:44-05:00

In my first post, I described the benefits of client side Javascript programming. In my second post, I highlighted some of the differences between React and other frameworks. Now It's time to get our hands dirty with React.

Setting Up

To follow along the example I'm going to provide, we'll use the React Starter Kit. Just click the link and download the kit.

We'll be making an example folder to hold our javascript and web page. Go ahead and make a directory, for example example and move the Javascript files from the build directory into that folder.

Next, make a simple HTML page with the following content:

    <!doctype html>
    <meta charset="utf-8">
    <html>
      <head>
        <title>Hello React</title>
        <script src="./react.js"></script>
        <script src="./JSXTransformer.js"></script>
      </head>
    <body>
       <div id="app"></div>
       <script src="./example.jsx"></script>
    </body>
    </html>

We'll be using example.jsx for our code. I strongly suggest you use a JSX aware editor such as Emacs or Vim.

Components

The core building block of React is that web applications should be built up of small, composible parts that React calls "components". For those of you familiar with GUIs, you can think of a component a bit like a widget, a single contained piece of GUI functionality. Let's dive in with a simple component that will take no arguments.

Let's say that we want to display the weather. Let's create a small component which we can use to display the current temperature. We'll call our component "Temperature", and we'll construct it using the React createClass method, which takes in an object and returns a React component.

 var Temperature = React.createClass({
    render: function(){
        return (
            <h1>The temperature is: 72F</h1>
        )
    }
});

We've hard coded the temperature into the component, but we'll fix that later. For now let's focus on the construction. We have created this component using the createClass method and we've given it a dictionary containing a single key, "render". The render method is what React will use when displaying our component.

When our component will render, it will display:

The temperature is: 72F

The component won't yet render on the page- for that we need to tell our component to mount to the DOM.

You may have noticed that we created a div with the id app in our HTML. We will mount our component at the app div. We didn't have to call it app, that's just convention. We could have called it temperature or demo or anything else, but app is a good convention to use for the top level of a React application.

To tell our React application to mount on the app DOM element, we'll add this to the end of our Javascript file:

React.render(
    <Temperature />,
    document.getElementById('app')
);

Now that we've done that, we can surf back to our web page in a web browser and we should see a plain page with our temperature prominently displayed.

Working With Props

Now let's replace that hard coded temperature with one we can set.

React components can have information sent to them in creation time through the use of what are called properties. Properties are constants which we can set at the creation time of the component and they're stored in the component's .props property.

Let's make that component again, but this time we'll get the temperature as a property:

var Temperature = react.createComponent({
    render: function(){
        return(
           <p>The temperature is {this.props.fahrenheitF<p>
        )
    }
});

Now, where we mount our component, let's set the Fahrenheit property. At the bottom where we mount the component, change <Temperature /> to <Temperature Fahrenheit=80 />.

Now save and reload the page.

You should see that the temperature is now 80.

Things are really heating up!

Working With Curly Brackets

Now let's imagine that instead of just Fahrenheit, we wanted to take our temperature and display it for people who live outside the US, in Celsius.

We can do the temperature conversion right inside our render method!

So let's go to our render method inside our Temperature component and change it to:

var celsius = (this.props.fahrenheit - 32) * 5 / 9;
return (
   <h1>The temperature is {this.props.fahrenheit}F/{celsius}C</h1>
)

If we save and go back to our page, we should see:

The temperature is 26.666666666666668C

Since that's not very pretty, we can use the built in Javascript rounding functions to make that a bit more pleasant to look at:

var celsius = (this.props.fahrenheit - 32) * 5 / 9;
celsius = celsius.toPrecision(1);

And now we should see:

The temperature is 26.6C

Much better.

This is not templating

At first glance, with these HTML tags and curly brackets it might look like we're using a templating system and then creating a string to pass into the DOM, but that's not what's happening here.

The HTML tags are not raw strings- they are getting pre-processed by the system and turned into function calls. These function calls are creating components, just like our components, which in turn get turned into raw DOM objects.

In addition, what's inside the curly brackets is not simply string expansion; it's fully evaluated Javascript that gets run during the render method's call.

The HTML-like syntax is called JSX, and it's use use optional but highly recommended. You can read more about JSX at the official documentation but the raw javascript code is run through a pre-processor and the XML syntax is turned into raw Javascript function calls, so a component that looks like <foo arg={val} /> turns into something like foo({arg=val}). For more information on how JSX works, you can refer to the official documentation.

Even though there is additional processing done during the pre-process phase, there's a huge runtime speed advantage of creating raw DOM objects rather than placing a string into the DOM. This translates into a faster application overall.

In addition, by using JSX, we are more able to model the output that we want, rather than worrying about the implementation of how the DOM objects are rendered to the browser, and if we use a JSX aware editor, our XML stands out from the rest of our code, making it easy to identify which parts of our code are related to rendering things on the screen.

What is this?

You may have noticed that in our render method, I referred to the Fahrenheit property as this.props.fahrenheit while I referred to our celsius variable simply as celsius.

That's because the Celsius variable was set inside the render method. The props, on the other hand, are a property of the Temperature class itself, and React uses the this property to refer to the object.

If you're familiar with Python or Ruby, this is somewhat equivalent to self. In Javascript, this will always refer to the function that's being executed, but this can be passed around from function to function through binding. By default, all the methods create by createClass have this bound, which gives us access to the properties of the object.

Next time

In the next post, I'll describe using state on objects, combining components and using other React component methods.

Modern Web Development With React (Part 2)

2015-02-01T07:19:52-05:00

In the previous post in this series, we discussed the history of web applications and the use of Client-Side MVC. In this post, we'll go through some of the existing Javascript client side frameworks and how React differs from these projects and why it's so versatile.

Before we dive into the specific web frameworks, let's quickly revisit the issue of client vs server side applications and logic.

Using a traditional web application framework, the web pages are created server side, then sent to the web browser, where they're rendered.

To create the web pages, most web frameworks use some sort of templating engine, which allows the pages to be constructed using composible parts. Most templating engines allow for constructs such as inclusion, looping, and an inheritance model.

Most modern frameworks also use some sort of Object-Relation Mapping (ORM) between the application and the database, and use some sort of mapping between application logic and the web server urls.

In the Client-Side model, we're moving most of this application logic up to the client, where it will live inside the web browser. Then, when needed, the client and server will communicate.

For example, if we were creating a shopping cart application, instead of creating a product page template and sending that to our user, the application would request product information from the server, and the server would return that data, usually in JSON.

Our product API call might look something like GET /products/123 and the server would return something like

{"id": 123,
 "name": "Wizomatic",
 "description": "It slices, it dices!",
 "price" 99.95
 }

which would then get turned into a web page.

This is roughly analgous to how an ORM object would get turned into a web page in a traditional web framework using templates, except with the advantages of rendering and caching outlined in the previous post.

While this series will focus mostly on React, it's worth mentioning that the are other client-side frameworks, some of which are older than React, and deserve some examination, such as Backbone.js, Ember and Angular.

This post will focus on the similarities of these frameworks, rather than their differences. For a more comprehensive view of the differences between these frameworks,t

Once we've established that simple templates aren't enough, we need to consider what to replace it with.

One of the first templates to answer this question was Backbone.js, then Ember and Angular.

These three frameworks are different (their differences are best captured in this article.

These frameworks share several fundamnetal concepts which differ from React. Firstly, they're all full application stack frameworks. Secondly, they all support the idea of data binding, and thirdly, they share a lot in common with the MVC design pattern.

React is largely focused on handling the display and interface parts of the application. That's different from some of these other libraries which cover not only display, but also cover things like object representation and client/server communication. React is agnostic to these concepts, and you're free to store your objects however you like, both on the server and on the client side.

At the same time, if you're coming from one of these existing libraries, you might at first wonder where a lot of the logic around object storage and syncronization will go. I'll cover that in a later post in this series, but the short answer is that the object's representation can be more flexible without the need for complex serialization/deserialization methods inside the application.

This dovetails into the issue of data binding, as data binding between server and client is a feature of other libraries. If you're unfamiliar with data binding in Javascript client libraries, it's the idea that if you have an object in memory on the client side, that the library will handle syncronizing that object's state with the server. React does not do this by default, but it is supported when explicity asked for.

Lastly, React does not impose the MVC design pattern. Since React is largely concerned with the View, how you choose to handle the other apspects of your design are left for the developer to choose. That's not to say that there aren't conventions and librariies which make this choice easier- there are libraries like Reflux (which I'll cover later) and Morearity, which users can choose from and find a design model that works best for them.

In fact, it's even possible to use Angular and then choose React as the view controller. While potentially heavy-weight, this is an option available to the developer.

This post has outlined some of the differences between React and other client side web development frameworks. In the next post, we'll roll up our sleeves and get our hands dirty.

Modern Web Development with React (part 1)

2015-01-07T14:18:12-05:00

In this series of posts, I'm going to present modern web development using React. As I do, I'll touch on the fundamentals of web development over time, the benefits of client side web frameworks, the React framework, then other tools in the React ecosystem.

Before thinking about React, let's first about how we would go about writing a web application without it. The most obvious way we can write a web application is that we can generate each page as a document on the server and that document from the server to the client. This is what we've done since the very beginning with Perl/CGI, and what we've been doing with frameworks like Ruby on Rails, and Django, and J2EE, etc.

In this traditional web framework model, the server keeps the application state. When a user performs an action, the web browser notifies the server and the server sends back a new document to display.

This traditional model of application development is well understood, but when we want our web applications to feel "responsive", it begins to fall down.

To address the performance issue, the answer for a long time was to use some Javascript to put some application interactions on the client, ie the web browser. Since the Javascript will run on the client's computer, it will appear faster. An early example of this that I remember is that the javascript would be aware of where on the page the mouse pointer was and would change the text color when the user was hovering over some portion of text.

With the advent of AJAX, the server could now send portions of data back and forth to the client. This meant that, for example, a post on a web forum could be sent and the user could submit it without leaving the page.

This technique for interacting between server and client grew increasingly sophisticated and over time, we've seen things change into the current era of web application development where the bulk of the visible parts of the application run on the client. These are often called "Client-Side MVC" web development.

Client-Side MVC has a number of advantages.

From a user perspective, a site that resides on the client is going to be faster and more responsive. Because the majority of the user interface code is running purely on the user's computer, the site "feels" more responsive. That's because instead of needing to request a page, wait for the server to generate the new page, send that new page and then display it, the web application can stay running while new data is either submitted to or fetched from the server.

In addition, because the user interface is separated from the data, the amount of data transferred between server and client can be reduced. Even if there's a large initial transfer of code (ie the application itself), each subsequent transaction between server and client can include just the data that's changed, rather than every visual element on the page. This means that even when data needs to be exchanged between client and server, there's less data to be transferred and the application feels more responsive.

Furthermore, keeping the data separate from the display means that the application can take advantage of caching and other techniques to reduce the need for client/server communication.

Lastly, using this technique can lead to more "interesting" user interfaces and functionality like auto-completion and rich graphical and textual interfaces.

From a technical perspective, the server code can be written more in a more straightforward manner. Instead of needing to process templates and generate pages, the server can support a simple API that is shared between it and the client.

Since all programming libraries are essentially APIs, programmers are already familiar with using APIs. Because the scope of a particular API is limited, these web APIs are easier to develop and debug. They can also provide a clean separation in development between frontend UI development and backend server development.

After developing sites with a client-side MVC framework, it's hard to think of going back.

In the next post, I'll discuss the various Client-Side MVC frameworks.

NYC Blooms with OpenStreetMap

2014-12-01T06:54:16-05:00

A community mapping project in New York is showing how OpenStreetMap can not only show how mapping can help people navigate, it can shape how people perceive their city.

New York City contains hundreds of community gardens. These gardens represent the the hard work of thousands of New Yorkers coming together to make these spaces come to life, and in some cases growing their own vegetables to be either given away or sold at local markets. Yet these gardens don't appear on most maps. Only a few of them appear on Google Maps, and none of them appear on the New York City maps, where the community gardens appear as vacant lots.

OpenStreetMap NYC, the local OpenStreetMap community group, in collaboration with GrowNYC, a New York City based environmental non-profit, are working to increase the visibility of these gardens through a collaborative mapping project. GrowNYC has released their data about New York community gardens to the OpenStreetMap community to help encourage their inclusion on the map. Because of OpenStreetMap's flexible way of describing features, mappers are able to capture not only the name and address of a community gardens, but also features like the hours that the garden is open to the public, and even whether or not the garden offers composting.

On November 22nd, OpenStreetMap NYC kicked off this collaboration through a meetup where local community members came together to work on mapping the gardens. Leading this meetup was Eric Brelsford, a NYC Mapper and founder of 596 Acres, a non-profit dedicated to land use and advocacy issues in New York City and Mara Gittleman from the NYC Community Garden Coalition, an activist organization working to protect community gardens in the city.

This collaboration presents benefits to both GrowNYC and OpenStreetMap. As Mara Gittleman explains, "Having an up-to-date map can help gardeners, researchers, and allies make the case for community gardens to funders and policy makers by showing their proximity to things like schools, illustrate where they might make up for lack of access to parks, quantify environmental benefits, etc." Unlike proprietary maps like those from Google, the maps provided by OpenStreetMap give direct access to the geographic data, allowing for all the benefits to researchers and policymakers.

Using OpenStreetMap also presents benefits for to the community gardens. Garden organizers and maintainers can now keep the information about their gardens up to date on their own, without needing an intermediary. And because OpenStreetMap is used by so many websites, these community gardens will be getting exposure to millions more people than they have before.

Because of its collaborative nature, its purposefully neutral stance on data and its extensible data representation, it seems likely that more community groups will use OpenStreetMap to shine a spotlight on previously invisible features of our world.

The Maintenance of Imported Data in OpenStreetMap

2014-03-13T09:10:06-04:00

Some feedback about my previous import post was regarding my contention that imported data is harder to maintain. This conclusion is based on years of observation. In this post, I'll explain why I believe imported data is not the same as manually mapped data in terms of maintenance.

The standard disclaimer applies here. I am speaking for myself and not representing any other group. In addition I want to say that this post is not meant to discourage imports in general, but that if OpenStreetMap is to have imports, we need to have comprehensive discussions about the issues around them.

Adding vs Correcting

The most common issue cited about why imported data is harder to work with is that it's easier to fill in an empty map. This often leads to the debate about completeness vs accuracy- in other words is it better to have a map that is full of partially accurate data or one that is only partially filled but with completely accurate data? This debate misses the real issue, which is the relative difficulty of adding vs correcting OpenStreetMap data.

On a map that is empty, it's easy to see what needs to be added. If a road or building is not present they can be added. We see this pattern being played out in OSM as larger, more prominent roads get filled in first, followed by secondary roads, and finally residential roads, buildings, POIs, etc.

The more difficult task is determing what data needs correction on a map that is visually complete. Looking at this map of Washington, DC, we see roads, buildings, and a variety of POIs. This map may be out of date. The speed limits on the roads may have changed, a building may have been torn down, or a store may have changed names.

It's not possible to determine what information needs correcting simply by looking at the map, nor is it always possible by looking at aerial imagery. Updating the map is far more difficult than creating it. Either the mapper must know the OSM data is wrong and re-survey the area, or they are going off newly collected data and the new data must be reconciled with the old data. For example we might know that a store is present on a particular street, but to be accurate, we must know not only that the new store exists, but that it replaced the old store in the same location.

When data is imported, the map appears complete, and the burden of the community effort shifts from collection to maintenance, which is a task that most mappers are less familiar with. In addition, correcting data does not have the same intrinsic motivation as first time collection does.

Complex Objects

The second issue that imports sometimes bring about are complex objects that are difficult or impossible to fix.

In this image, we see a section of New Jersey in OpenStreetMap, with the patchwork of color representing landuse data

When we zoom in, we see one of these objects is a multipolygon relation, an object composed of several other objects. This one object references five other objects. In addition, the outer boundary of this woods shares geometry with about another ten landuse objects.

If someone were to try to update the landuse data, they might unintentionally be effecting up to a half dozen other objects at the same time, meaning that a simple update process now becomes a difficult and time consuming process of ungluing multipolygons, splitting them and trying to reconstruct them.

For an experienced mapper, this would be a tedious, time consuming process. For a new mapper, it would be an daunting and possibly confusing challenge. Most mappers simply avoid the work altogether and the data is never updated.

Unverifiable Objects and External Unique Identifiers

The third, and in my view the most common problem we encounter with imported data is the issue around external identifiers in the imported dataset.

Let's start with a simple example of the problem in action: New York City bike racks.

As you can see, there are two bike racks on this street according to the data.

If one of those racks were found to be missing or in the wrong location, it would need to be removed. The problem is that it's not obvious by looking at a bike rack what bike rack is what. What, then is the solution?

Taking another example, we see this building footprint from New York City:

Every imported building in New York has a Building Identification Number, or BIN. In OpenStreetMap we've separated out the garage from the building, whereas the city hasn't. When merging the city data with data from local mappers, do we apply the BIN to the garage? To the main building? To neither?

A more common example is data imported from GNIS. GNIS data was imported many years ago and has many known issues associated with it, including but not exclusively that many of the GNIS points are far from the actual location of the object, in some cases a half a mile away.

Taking a look at this node, just a few weeks ago, this had a GNIS datapoint for a school. Korzun, a very active mapper, visited the location himself and found out it was a drug rehab center, but was too concerned about the integrity of the GNIS import to remove the dataset, even though he knew it to be inaccurate.

If an experienced mapper like him is confused, imagine the confusion of less experienced mappers?

The end result in all of these situations is out of date, or entirely inaccurate data in OpenStreetMap.

To address these issues, I have three practical suggestions.

We should reduce or eliminate the use of external tags, including identifiers during imports

For years, importers have been bringing in external data along with the imported data. This data might include collection dates, unique identifiers, etc.

As part of the import review process, OSM has generally been reducing the import of much of this data (for example, collection dates), but we continue to import external identifiers. The idea behind including the external identifiers is that it's hoped that at some future time, we'll be able to do an update based on a new dataset and use the external identifier as part of that process.

Unfortunately, while this possibility has been discussed a number of times, in the nearly ten years of the project, no one has successfully been able to make this happen. The reason is that even if an external identifier is consistent across revisions (which is not always the case), OSM activity does not lend itself well to using the external ID as the only identifier in the merge process.

OSMers may decide to add features that are not present on the map between imports, or may decide that a feature which has a single identifier in the external dataset should be represented in OSM by more than one object (eg splitting a road, or separating two buildings connected by a skybridge). It's even possible that a mapper will accidentally modify a tag that they do not understand. This was quite common when bot-mode expanded names across the US. Many times it was unable to fix a road simply because the TIGER tags had been edited by a mapper who did not understand them.

Because of this, merges must take other factors in consideration when doing a merge, including the feature name and location. While slower, these methods prove superior during a merge- leading back to the original question of why use the unique identifier in the first place if it creates confusion (leading to stale data) and does not offer benefits in the long run.

Imported data must always be imported by hand, and not by script

When imports began, the process for getting the external data into OSM was that the data was converted into OSM and then simply placed into OSM. As we discussed earlier, it is easier to add good data than correct bad data so we should take advantage of this in future imports.

We have tools to make this easier, including the OSM Tasking Manager, which is being used in such an import now. Updating OSM this way not only gives an import more review, but can also be used a means of engaging users, as was shown in the Seattle import.

An import proposal must be accompanied by a means of updating the data in the future

OpenStreetMap is only valuable if the data on it is not only complete, but current, and we've seen through OSM that data from imports is not often updated. We need to change this, and the best way to do it would be to require that any import specifically address future updates.

If possible, that process will not only involve conflating the data from the external dataset with OSM, but continuing to keep OSM contributors engaged in OSM editing.

Conclusion

Imports have their place in OSM, but they are not without their perils. I've tried to outline specifically how the past and current import process does not serve either data consumers nor the OSM community in general, as well as provided concrete suggestions on moving forward. I hope that we can continue to make OSM not only the freest, but the best geographic dataset in the world.

Esperanto After Four Weeks

2014-02-13T09:38:48-05:00

I'm into my fourth week of Esperanto and I thought I'd write some reflections on the process, the struggles and successes.

The Methods

My main method of learning Esperanto is the site lernu.net and its "Bildoj kaj demandoj" (Pictures and Questions) course. In addition, I've gotten some help from Esperanto speakers, and I've been listening to Esperanto music from Muzaiko, as well as the Esperanto-USA podcast.

Lastly, when specific questions arise, I've been using the 16 Rules of Esperanto Grammar, the English-Esperanto Dictionary, and on occasion, simply using Google Translate.

Quantity

I don't study Esperanto every day, but I do lessons about three or four times a week, at least. The amount of time I spent depends on how quickly I feel I'm absorbing the lessons. If I'm understanding the lesson well, I do more, and if I'm having trouble, I go slower.

Process

For me one of the more illuminating experiences has been in seeing how my own mind handles the new material. Esperanto is quite easy, but occasionally I find myself having difficultly with the material. Lernu has been an incredible resource for that. Using Lernu, I'm able to set my own pace, learn as much as I can, and then put the computer away. When I'm ready to take on the next lesson, it's there for me.

In addition, after each section, the software gives small quizzes. Each quiz question covers material in the previous lesson. If you struggle with a question, the software offers you the opportunity to see various ways the question can be answered.

In order to get the feel of the spoken language, I listen to Esperanto music and spoken Esperanto in the podcasts. In order to get the "mouth feel", I try to speak every section out as I go along in the lessons. Lernu provides audio for every question/answer pair, which makes it easy to repeat as I go along.

Results

I've worked my way through about two thirds of the basic language course.

In just four weeks of learning Esperanto, I can pick up on about 20% of the Esperanto USA podcast. It's not enough to get the gist of the conversation, but it's enough to pick up meaning here and there. In terms of reading, I'm at about the same level. I can figure more from context and the use of a dictionary, but it's very slow.

On the positive side, I feel that I've been able to learn at a very steady pace and I feel comfortable learning more, which is good because there's a lot more to learn.

Meta-learning

In addition to just learning Esperanto, I've learned a lot about how I learn languages, which was a big part of this experiment.

Spelling is a major issue for me, even in English, and Esperanto hasn't been an exception. Even though Espernato is entirely phonetic, I often make simple mistakes, spelling "ĉi" as "ĉe", or "li" as "le", or transposing two letters.

Vocabulary has been a mixed bag. Many of the words in Esperanto share a root either with English or Latin, making them easy to remember. Some words have no equivalent, and those words require work to memorize. A technique that I've found very helpful is to use either a mnemonic device to help me memorize a word until the word can "stand on its own". For example the word for "when" is "Kiam", and I remembered that by knowing the root "ki"- followed by "am", which reminds me of "AM/PM". I was struggling with "kvar", the number four, and so I visualized the band Gwar and imagined that there are four members. The fact that there are actually five members of Gwar is unimportant, since it's the connection that I'm trying to build. For the word "mateno" (morning), I associated it with the English word "maternity", which I associate with birth, visualizing the day being born. And lastly, I remember the word "pilko" (ball) by thinking of Karl Pilkington.

The Lernu site has tests, and a majority of the questions that I've gotten wrong have been spelling errors, followed by forgetting the correct tense of a word, followed by forgetting a word altogether. This mirrors my experience in English, where even today, I misspell words or forget a word in the middle of a sentence occasionally.

Conclusion After Four Weeks

I've been really happy with this experiment, both in learning Esperanto, and learning about my own learning process. I think the Esperanto is going at a speed I can expect based on the time I put into it, and as I go, I'm learning an enormous amount about myself in language acquisition.

Why Imports in OpenStreetMap Are Controversial

2014-01-25T21:05:35-05:00

OpenStreetMap's goal is to map the entire world, so one might assume that anything that would help give the project a leg up would be welcome, but what many potential importers find is that the OSM community, in particular the more senior members, are hostile to imports. Understanding their strong feelings can be difficult, but I'm going to take a look at the issues around imports in the community, and why they're so controversial.

Before I go on, I need to make clear that the views in this post are my own and don't reflect the views of the DWG, the OSMF, OSM US or any other organization.

OSM's Construction

OpenStreetMap is a single unified geographic dataset. As a consequence of its design, it's far more integrated than even professional GIS maps are. For example, you can't simply remove all the roads objects in OSM, because those road objects might be connected to other objects, such as political boundaries, or natural features. Most professional GIS maps use a system of layers which are placed on top of one another. In contrast, OSM maps are more like a complex weaved material, with geographic features intertwined with each other.

This makes some operations in OpenSteetMap very easy. If a political boundary runs along a river, for example, as the river becomes more detailed, so does the political boundary. At the same time, it poses a set of challenges for imports.

Licenses

The most common problem with a data import into OpenStreetMap are licensing conflicts. In an ideal world government datasets would be released into the public domain. Unfortunately this is the exception rather than the norm. In the United States most states and municipalities do not release their geographic datasets to the public without restrictive licenses and expensive license fees.

Some municipalities are trying to make their data more available but do not use standard licenses. Instead, most US municipalities create their own licenses from scratch, oftentimes with conflicting terms. It's not uncommon for a municipal website to say that the data is in the public domain in one part of the website, but then another part of the website will say that the data is under copyright and may not be used or distributed without explicit permission.

The situation in the US mirrors much of the rest of the world, where national or local governments struggle between "openness" and "control".

The result for OpenStreetMap is that getting a license situation clear is difficult or sometimes impossible. Because of the way OSM data is so heavily integrated, removing it afterwards is nearly impossible, so the project needs to be extremely careful beforehand about the license situation for any imported data.

Official vs Authoritative

The second challenge or imports is the issue of official vs authoritative data. Most people assume that if a dataset is official that means that the data is of extremely high quality. This can be true, but more often than not, the official dataset is not what one would expect.

When a government creates or maintains a dataset, it does so with a specific purpose in mind. For example, in the US TIGER dataset, the purpose of the data is to provide a mapping of roads in order for census employees to know where to look for houses. The TIGER dataset needs to be complete enough for the census takers, but that is not the same as being a comprehensive road network map. For example, if a house exists at the end of the road, it does not matter to a census taker if the road has a slight bend in it or not.

Similarly, other data sets are often created to meet a very specific need. In order to meet that need, a local government will often outsource the task of surveying, or will hire low wage employees to do the data collection. The results of the survey need to be consistent, and governments often provide exhaustive procedures and checklists, but only for the specific objective they're trying to meet.

This difference in mapping objectives often comes out in address data. A government dataset may provide addresses as points on a map but it's often unclear what these points correspond with. Do they correspond with the geometric center of a building, or do they correspond with an entrance? If they correspond with an entrance, what do they do in cases where the building has multiple entrances?

In practice we find that such datasets are often complete in that they show all the addresses, but are inconsistent in where the address points are placed across buildings.

This issue of accuracy and consistency becomes more pronounced on larger, national datasets. A national dataset often consists of a collection of regional datasets, each which have been collected by different individuals and organizations, leading to inconsistency. Paradoxically the national datasets are the ones that OpenStreetMap members are most interested in, since they can provide the most information.

Doing it Right

Once the issues of license and data quality are addressed, there are questions of how to get the data into OSM. As mentioned earlier, one can't simply add external data onto OSM- the data needs to be integrated.

This integration consists of many highly technical, highly detail oriented steps. Every new importer believes that the integration step will be easy, but the fact is that the process is difficult and tedious.

OpenStreetMap is full of very badly imported data, and suffers from it to this day.

Corrections and Updates

Unfortunately, even after nearly a decade, no one has found an ideal solution to this problem. The result is usually that data imported into OSM never gets updated and the imported data in OSM becomes wrong very quickly.

When considering updates, as discussed previously, we cannot assume that an administrative database is correct. Ideally we want OSMers to map the areas themselves, and it's inevitable that they will do so, leading to a conflict between the two datasets.

OSMers have oscillated between trying to update data based on government datasets and always assuming that local changes are correct. Neither of these techniques will work in every case. Instead, a mapper will need to check the conflict manually, preferably by manually surveying the area.

This process of updating is slow and tedious and has only happened in a handful of cases.

Community Helping or Harming

Within OpenStreetMap there is a debate on whether or not imports help or harm the OpenStreetMap community.

OpenStreetMap works because its members are constantly adding and correcting information in the system. If there were no community members collecting this data, the map would quickly suffer bitrot, where old data sticks around and is not replaced with current, up to date geographic information.

Because of that, some members believe that efforts should be focused not on bringing data in from external sources, but on growing the community of mappers. Others believe that "seed data" provides a starting point for more mapping, especially in places that have not been very active in OpenStreetMap previously.

I'm not going to rehash these arguments here, but I will say that the role of imports in OpenStreetMap remains contentious, and an area of research that I would like to see explored is the rate of update between manually surveyed and imported data.

Conclusion

OpenStreetMap only works because of our community of many thousands of dedicated mappers who spend their time updating our map. Whatever the role of imports today or in the future, they can never be a substitute for on the ground mapping, which needs to remain a cornerstone of our efforts in OpenStreetMap.

Fearing French and Embracing Esperanto

2014-01-23T14:01:10-05:00

This post is about why I fear French, and what my attempt at taking this problem on has taught me. This is one of the most personal blog posts I've made in several years, but in it are lessons for nearly everyone, especially if you are considering learning a second language

[UPDATE 01-25-2014]: There seems to be some confusion about my use of the term "polyglot" in this post. I am not equating programming languages with spoken languages. I am comparing the introduction of a new skill in an existing domain. Also, I'm referencing a talk by Adam Jacob, a well known member of the system adminstrator community about how we system administrators are "polyglot programmers".

My Background

I was born just outside of Paris in 1978. My father was French, and my mother is an American who went to Paris to perfect her art and ended up marrying a Frenchman. They intended to spend the rest of their lives in Paris and so when I was born, they decided to make English the "home language"- thereby ensuring my fluency.

When I was just over two years old, my parents moved to the United States, and while I had been absorbing French, I wasn't any more fluent than any other two year old. English remained our home language in the US.

French was spoken in my home, but it was never spoken to me. My parents used it to communicate with each other as a sort of secret code. They did this in delicate situations outside, but they also used it to argue in front of me so that I couldn't understand them.

The attitude my parents took was that I should learn French- but that I should do so on my own. My father recorded French in Action off the television for me and asked me to watch it.

Every few years, I would return to France with my family and be reminded by my grandparents and uncle that I should speak French, and that it was a shame, or a failing, that I didn't.

I took a French class in college, and it also used French in Action, and I learned only marginally more than I had before, from watching the course on television- which is very little. While I could understand the lessons, the structure of the quizzes and the time allotted meant that I had virtually no time to figure it out. I'd misread or misunderstand a question, get the first five answers wrong, then I'd figure it out and get most of the next ones correct. But the professor never took that into consideration, and I failed each quiz. Each failure meant each time, I was increasingly deflated, and eventually I fell behind. Even if I had aced the final exam, I would have failed the course due to the quizzes.

What makes this tragic is that I am, in fact, a French citizen.

And as a French citizen, I'm required to fill out paperwork in French. Years of anxiety and shame have left me with a feeling of dread on the topic. Even writing this out, I feel my jaw clench and the tips of my fingers become numb.

The Polyglot

Switching gears, as Adam Jacob would say, I am a polyglot. I do not speak any other natural language, but I able to program competently in four programming languages, as well as another half dozen with some help.

In addition to programming, I was a system administrator for over a decade, where it was my job to work on vastly different systems and in some cases make them communicate with each other.

How could it be that I could excel at quickly understanding and working with computer languages and systems, but with spoken language be nearly crippled. Certainly this was indicative of something being wrong.

I needed to figure out what the connection, or more accurately the disconnection was between these two skills.

Mastering a spoken language and programming a computer are entirely different, but the essence of my search is not comparing the two, but rather comparing learning an additional programming language with learning an additional spoken language.

The programming languages people may be familiar with are things like C, C++, Java or Python. All of those languages are popular, and all of them are similar. But there are other programming languages too, such as Javascript, Clojure, Haskell, Erlang and Prolog. You may have heard of Javascript, but it's less likely that you're familiar with the others. Programming in any of these languages is different than programming in Java. Each requires learning not just a syntax, but a change in programming style altogether. Each of these languages will teach you a new way of interacting with the computer and a new way of thinking about computer programming.

Of the languages listed, I only know Clojure and Javascript, but I'm confident that I could learn any of the languages I mentioned because I've made the leap in the past. I've learned new programming languages successfully, even when they required a major change in mindset.

Rethinking Language Learning

How could I transfer my computer language skills to spoken language?

There were lots of TED talks on this very subject.

Chris Lonsdale says you can be fluent any language in six months, and Conor Quinn says I can start speaking in 10 days.

Benny Lewis talks about hacking language learning and Sid Efromvich has 5 techniques to speak any language.

If you're interested in language acquisition, I suggest watching all of these talks, as they each teach something different and complementary, but what they had in common was:

Communicate as early as you can, even if you aren't speaking
Don't be afraid to make mistakes, because it's inevitable
Learn/speak in a way that's relevant to you personally
Retrain your ears and your mouth
Language acquisition is a skill
Language acquisition is a skill that's best learned when we're mentally at ease

For me, those last two were the most important.

Knowing that language acquisition itself is a skill changes the process from "Learning French" to "Learning Language Acquisition", and all of those years of shaming, whether by family or academia, helps explain why that process has been so difficult.

There were two ways to approach this problem. The first was to try to emotionally move past the issues I've had with French and try again, the other option was to learn an entirely different language in order to gain the skills of language acquisition and then apply those skills to French. As you might guess from the title of this post, I opted for the second and decided to learn Esperanto.

Esperanto

The natural question at this point is either "Why Esperanto?" or possibly "What is Esperanto?"

Esperanto is a constructed language, in other words it's a language which was created by a person or persons intentionally, rather than having developed naturally over hundreds or thousands of years as most languages do.

I'd known about Esperanto for many years. It's mentioned casually in the BBC show Red Dwarf, where all the ship's signs are posted in both English and Esperanto. I'd also heard about it because more than a few students from Central and Eastern Europe that I knew spoke about it, but I didn't think anything of it.

It's only now with the convergence of my interest in learning French and my reading Land of Invented Languages which has sparked my interest. As a practical exercises in language acquisition, Esperanto comes with some very impressive credentials.

The aforementioned Benny Lewis claims that just two weeks of Esperanto can get you months ahead in your target language, Esperanto USA claims that Esperanto is four times easier to learn than any other language, and Wikipedia has an extensive article on the foreign language benefits of Esperanto.

This idea is summarized well by this talk by Tim Morley entitled Teach Esperanto First, where he describes the process of teaching children Esperanto in order for them to gain language acquisition skills, applicable in any language.

This idea of a simplified first language resonated with me, just as I think that Python is a great first programming language.

There are many resources for learning Esperanto online, including and especially lernu.net.net, which I have been using. In addition, there are books online, including audiobooks at LibriVox.

Why Esperanto is Easy

I've been using the Lernu online course for about two weeks now and I have to say that it's been wonderful. Along with some basic understanding of the Esperanto grammar, I've been able to learn an incredible amount in a very short time. The language is extremely elegant in that every word is constructed from a root word and then modified using a series of simple prefixes or suffixes.

From a vocabulary perspective, Esperanto is also very easy as most of the words have a Latin, German or English root, and because English is composed so largely of French and German roots already, it makes acquisition that much easier. For example the word for dog is hundo (nouns all end in -o), the root being the German "hund", which in English became "hound", and so on.

Between the partially familiar vocabulary and the simple rule based word construction, understanding Esperanto is fairly straightforward. Obviously mastering any language takes time, but Esperanto gives the learner a step up at each hurdle.

Even the simple words have hints about their meaning in them. The word kio such as in the sentence "Kio estas tio? (What is that?) vs the word kiu, such as in the "Kiu estas tiu?" (Who is that?). The -o at the end of kio indicates that the question is a "What", which is the same as the -o ending used by all nouns. The -u indicates a pronoun question, "Who". But just by remembering the -o noun rule, one can differentiate the two words.

The phonetics of Esperanto are fairly easy. Each word is spelled phonetically, with no silent letters, two letter combinations or verb modifiers. The sounds used in the language are relatively familiar to English speakers who've listened to European languages and are easy to pick up.

Esperanto Challenges

There are a few challenges with learning Esperanto. Because it's an auxiliary language, there are fewer people using it every day for one on one interactions and so you see less original culture in Esperanto than you do in a natural language.

In addition, because Esperanto is usually learned as a second language, it's slightly more difficult to train your ears and your mouth to the language because each Esperanto speaker has a slightly different accent based on their linguistic or geographic background. Even the few native speakers purportedly have slightly different accents. This is somewhat offset by the fact that most Esperanto is spoken relatively slowly in the various recordings that I've heard.

All in all, the downsides issues are minor compared to the ease of language acquisition that Esperanto provides, and there is a lot more in Esperanto than in other constructed languages such as such as Lojban, Ido or Interlingua.

Learning Esperanto

If you want to learn Esperanto, I recommend the course I'm taking at Lernu, followed up by checking out this amazing resource of links to travel groups, books and podcasts in Esperanto from the Esperanto club at the University of Rochester.

I have a long way to go until I'm fluent in Esperanto, but I feel that with a new framework for examining language learning as well as the experience of Esperanto under my belt, I will be in an excellent place to not only speak with other Esperanto speakers, but to be able to tackle my fears of French.

Ĝis la revido!

Edit Wars in OpenStreetMap

2014-01-17T22:12:07-05:00

Reading the feedback from my Why The World Needs OSM post has been enlightening. One criticism of OSM that has come up multiple times is the concern about edit wars like those seen in Wikipedia. It's a topic that I know something about, both as a long time OSM contributor, and also as a DWG member. This post explores how common edit wars are and the circumstances that surround them.

Before I go any further on this topic, I want to state for the record that the opinions expressed in this post are my own and don't reflect the DWG or the OpenStreetMap Foundation in any way.

OpenStreetMap DWG members operate similarly to Wikipedia Administrators - we handle issue of users breaking community standards and intervene in issues of conflict resolution. We don't enforce any particular mapping conventions, but we do enforce community guidelines.

What are the kinds of editing conflicts seen in OSM?

The most common conflict is around the issue of data imports. OSM is not a collection of information sources, but a single unified dataset where geographic objects interlock. Because of this, and because of OSM's emphasis on primary sources and community mapping, imports are very difficult to do right. In response, OSM has a set of import guidelines which must be followed. When they are not, the DWG steps in and enforces that the guidelines are met.

The second most common place that the DWG steps in is vandalism. Vandalism is purposefully editing the map in a way that doesn't reflect the reality. OSM community members often fix small mistakes by newbies or small scale vandalism. The DWG often gets called in to fix larger scale vandalism, usually by drive-by troublemakers.

The third most common type of case that the DWG handles are around tagging conflicts. Tagging is the process of classifying objects. OpenStreetMap uses a free form folksonony and sometimes disputes arise about how to classify certain features. I hesitate to call this an edit war because it is almost always a single user whose views differ from those in the rest of the community.

What about actual edit wars?

Actual edit wars do occasionally come up, but they almost always involve political disputes. One group believes that an area of land belongs to them, and or has a certain name, while another group believe that they own the land, and or it has a different name and the two sides cannot come to consensus. In this special case, the DWG has special policies laid out from the Foundation in how these delicate situations are handled.

When an edit war happens, the DWG gets involved, but the frequency of a genuine edit war is very low, on the order of one edit war every six months or longer. It's far less frequent than what someone outside the project may imagine.

Why are edit wars so common in Wikipedia, but so uncommon in OpenStreetMap?

Wikipedia edit wars concern two issues, facts and presentation. In particular, Wikipedia relies on secondary sources. When two facts collide, an editor must decide which source is better. OpenStreetMap relies instead on primary sources of contributors visiting an area and editing OSM to reflect what they see. Secondary sources are considered inferior and many OSMers use the derogatory term armchair mapping in reference to using these sources. Because of this, when a conflict arises, the person who visits the location is going to be given deference over any other source, even officially published government data.

As for presentation, Wikipedians may argue over the best way to write a paragraph, or where to list a particular fact. In OSM the general rule is to update wrong information, and to add detail when you can.

I think that our policies around ground observable truth are the main reason why edit wars are infrequent and almost always concern political boundaries, which are the one feature that OSM maps that is not ground verifiable.

The other reason that I believe edit wars are less common in OSM is that OSM has a higher bar required for editing. OSM does not allow anonymous edits, and unlike Wikipedia. If a particular user is causing problems, they can be easily identified.

Additionally, OSM is quite a bit smaller than Wikipedia. Even though OSM has over 1.2 million user accounts, on a daily basis only about 2000 users ever edit the map. While 2000 is certainly not a small number, it's relatively easy to spot a troublemaker. But even then, a single user cannot start a war- someone who changes large parts of the map on their own is an isolated vandal, while actual edit wars would require at least two parties. This virtually never happens.

When another editor makes a change to something I've created, in every case I've seen, they've enhanced the map. Either they've expanded on the data that I entered or they corrected a mistake I made. This is the norm in OpenStreetMap- additive and corrective edits. Edit conflicts are so infrequent that most users will never encounter them, and even to the group that oversees them, they're a special case. It's possible that as OSM grows that the number of edit wars will increase, but I don't think that's likely, as we haven't seen the number of edit wars grow despite the project's growing success over the years. Instead, I think that the source of edit wars, that is the number of geopolitical conflicts is going to be a better predictor of the trend.

The Role of the Fringe

2014-01-14T08:15:27-05:00

I'm in the process of reading In The Land of Invented Languages- an excellent book I will write more about. For this post, I want to focus on how Esperanto's community is like our Free Software/Free Culture community and the role of the fringe element.

Reading about Esperanto and the struggles it's had over the years to be taken seriously in spite of a "fringe element" who are both exuberant in speech (and dress) as well as philosophically motivated reminded me of the struggles the Free Software movement. In fact these two movements are more similar than dissimilar in many ways. The answer to why this is this touches on the very core of both communities, which is the social cohesion between people who believe that they are changing the world.

The Free Software community has without a doubt changed the world. The Internet as we know it is built on and with Free Software. Billions of people use Free Software every day, either directly or indirectly, and Free Software has changed the dialog about the underlying assumptions our society holds about issues of morals and economics and how those underlying assumptions shape our actions and our laws, both as a group and as individuals.

Esperanto speakers set out to change the world by creating a unified language that cast off previous cultural influences. Esperantians wanted a neutral language in order to share their culture to the world using a dialect that wasn't shaped by any particular kind of thinking.

Both movements have been shaped and proselytized by those espousing its practical benefits over its philosophical ones. In the Free Software community, we have the Open Source movement, which has arguably overshadowed the older Free Software movement. In the Esperanto community, similar arguments broke out amongst its members.

In both communities, community members and community leaders complained about "the fringe", those members, often men, who didn't take their grooming too seriously, who dressed in odd outfits (in the case of Esperanto, lined with green stars- the Esperanto symbol), etc. Anyone who has spent a significant amount of time in either community probably knows what I'm talking about, and have had the same thoughts along the lines of "How can we be taken seriously with these people?" If you haven't had that thought, then you're probably one of the people everyone else is thinking this about.

The answer is tricky. After seventeen years in the Free Software community, I know without a doubt that these people have scared others away. In particular they've scared away people who weren't affluent, white, eccentric males. But on the other hand, they're the ones who still come to all the events and who speak publicly about the Free Software movement's philosophical underpinnings.

With any peaceful group that is trying to change the world, maybe these people are inevitable. The question for the rest of us is how to grow the community to be large enough that others aren't scared off, and that everyone has a home.

The Scoop on Cat Litter

2014-01-12T11:09:35-05:00

It's time to take a break from the high minded to the mundane. Today I'm doing a review of cat litter!

I have two wonderful cats, one male and one female, both about one year old, both about 7lbs, and both adopted from the ASPCA.Since I've owned cats as an adult, I've been searching for the perfect cat litter. I've tried six different litters so far, with varying results.

My regime is to scoop the cat box once a day, and replace the litter about once a week, adding more cat litter as necessary to replace what's been used up. I have a very large cat box, which can easily hold 20lbs of clay litter.

Living in Manhattan, there is a premium on space, and so a cat litter needs to do excellent odor control, more than it might if I lived in a larger space and could put the cat box in a more out-of-the-way spot.

Feline Pine

The first cat litter up is the Feline Pine. This is what the local ASPCA uses, so I thought it would be a good idea to use the same product during the cats' transition. This is a dust free, chemical/fragrance-free product made from wood.

Pros: I like that this is an all natural product, made from a renewable resource. It also scooped/sifted well.

Cons: The product smelled a bit like used cat litter (ammonia) to me even when just out of the bag. I thought the smell would dissipate but it just saturated the apartment and I had to get rid of it.

It also didn't do any clumping, although the company apparently also makes a clumping version.

Overall: 2 out of 5. Of all the products, this one irritated me the most even when brand new. I like the idea that it's all natural, but ultimately the product needs to be something I can live with.

World's Best Cat Litter

The next product on my list is World's Best Cat Litter, another all natural (corn based) product. I've met several people who use World's Best and it seems to have a good reputation.

Pros: The biggest advantage to World's Best is that it's flushable. You can simply scoop, sift, and toss out kitty waste along with your own. It won't clog or damage your pipes. It's also all natural, uses no artificial chemicals or perfumes. It also claims to be longer lasting than other products.

Cons: Firstly, and most importantly, I was unhappy with the odor control. After a few days, the ammonia from the cat box smelled throughout the apartment. Secondly, the litter tracked throughout the entire apartment and were a mess to find/clean up. Thirdly, the urine clumps sometimes crumbled on their way to the toilet.

Overall: 2 out of 5. I wanted very much to like this product, but it disappointed me. It worked better than Feline Pine, and it is environmentally friendly, but the odor control and tracking issues need addressing.

Yesterday's News

The third product on the list is Yesterday's News. This product is different than the other products in that it's made from recycled newspapers, formed in pellets, rather than in small granules like the other litters.

Pros: This product uses recycled newspaper, which is good environmentally. It's also dust-free, just like the World's Best and Feline Pine. It's also completely odorless.

Cons: This product does a decent job of odor control for a few days, but after that, the urine soaked newspaper pellets at the bottom of the litter lose their appeal quickly. The product is also not siftable- the pellets are too large. One of my cats cries when he walks on the product, and after accidentally stepping on a few stray pellets in the apartment, I cry too.

Overall: 2 out of 5. I love the idea of using recycled newspaper, and it does provide some odor control, but not nearly enough to make up for its inability to be scooped and sifted.

Fresh Step Unscented Clay Litter

The online reviews suggested that Fresh Step is the best cat litter available, but people complain about the perfume, saying it has a chemical odor. I decided to try the Fresh Step Unscented Litter.

Pros: Odor control at last! Of all the cat litters I've tried, this one does the absolute best in controlling odor. Even when cleaning/sifting, it reduces the odor to a very controllable level. It's easily siftable, it clumps fairly well, and it doesn't track as much as some of the other litters.

Cons: Firstly it should be noted that clay cat litter is mined, and is generally not considered environmentally friendly. On a personal level, this product is particularly dusty- the most dusty of all the products I've used. Pouring it out or sifting it releases a cloud of cat-litter dust into the air each and every time. Additionally, it does have a tendency to clump on the bottom of the pan and need to be scooped up forcibly.

Overall: 3 out of 5. If it weren't for the environmental damage of this litter, I'd rate it at least a 4, but it's created through strip mining. In addition, I'm not thrilled with the dust that it generates. Other than that, this is my favorite litter to use and until I find something better, is the litter I will have on standby due to its odor control and siftability.

PetCo Refillable Unscented Cat Litter

Once I tried the Fresh Step, I thought about cheaper alternatives. I am not generally a fan of PetCo due to their live animal policies policies as well as some horror stories I've heard about their store in DC, but there's a store across the street from my apartment and they offer a refill program where you can buy a box of cat litter, then refill it for less than the price of a new box. I decided to give the

Pros: This product is very inexpensive. It's the cheapest of all the products I've used, and the cats don't seem to mind it. It looks and feels very much like the Fresh Step, though less dusty.

Cons: It doesn't do as good a job at odor control as the Fresh Step. After about 3 days, despite daily cleanings, it smelled. It's not as bad as some of the other litters, but it's a significant difference from the Fresh Step. Additionally, it's another clay litter, with no indication that it's not made from the strip mining process.

Overall: I'll give this product a 2 out of 5. Like the Fresh Step, it loses a point for being environmentally unfriendly, and it looses an additional point for being bad at odor control, but it's also less dusty and cheaper than Fresh Step. I've had to knock another point down for finding out that this product is just regular bags of cat litter dumped into a bin.

Conclusion

I still have another half a container of the PetCo cat litter, which I will use up in the next week. I also want to try a silica based cat litter, one of the "Natural" labeled Fresh Step products, the "Advanced Formula" World's Best Cat Litter, and the Blue Buffalo Cat litter, so stay tuned for an update in a couple of months.

If none of these products works out, I will use the Fresh Step product since it's the best at odor control. In the meantime, if you're a cat litter company who wants me to try their product, don't hesitate to contact me!

How Sirius/XM mis-handles customers

2014-01-11T07:34:52-05:00

I had a recent experience with Sirius/XM that I think is a great lesson in how not to handle your customers.

Let's face it, Sirius/XM has an uphill battle in terms of entertainment. They have dozens of music channels, but a subscription to Pandora or Spotify will take care of all your music needs. What's left, and what's unique about the service is the premium content, the things you can't hear anywhere else.

+1 Sirius: Offer me a product I want

And that's why I enjoyed the service, for stuff I couldn't hear anywhere else. But the service was a pain in the butt to use. I didn't own a radio, so I had the Internet only service. But unlike a radio stream from SomeFM or Live365, or even my local Pacifica station, I couldn't just tune in and listen. I had to use a complicated web app on my computer, and I couldn't keep it on in the background because it would sign me out automatically after a while.

I could use an app on my phone, but I couldn't listen to Sirius on my Roku player, my Samsung Smart TV, my Boxee box or my homebuilt XBMC box. Somehow, three separate video services are able to deliver thousands of TV shows and movies to me, but Sirius can't deliver audio to me. Roku has sold over five million units, but Sirius still seems focused on the car market, rather than offering entertainment anywhere.

-1 Sirius: Offer your product as many places as possible

I mentioned earlier that Sirius has music, but it also has talk services from NPR, PRI and the BBC. But like music, all of those services offer live streaming, or better yet, podcasts. With podcasts, I can listen to the shows whenever I want, stop them, rewind them, or even share them with friend- all for free.

With the Sirius/XM premium content, I can't even choose to stream the content when I want. By comparison to modern content delivery systems, Sirius feels like a step backwards.

-1 Sirius: Offer the product in a form your customers want it

Because Sirius is such a pain to use, I simply forgot to turn it on for about six months. Every month, I dutifully (auto)paid my bill. But after my credit card changed, the autobill stopped working, and I received a call at 6pm someone saying they were from Sirius/XM and wanting to talk to me about my bill.

I told her that I didn't have time to talk at the moment, but that I didn't want to use the service anymore, so I'd like to cancel. She said that she'd done it, but I still had the matter of my outstanding bill. I told her I'd take care of that online.

She ended the conversation with "I'll call you tomorrow." I told her I didn't want her to call tomorrow- that I'd take care of it on my own.

The next day, also while preparing dinner, I received a call from a different gentleman claiming to represent Sirius/XM, and wanting to talk to me about my bill. This time, I'd had enough, and just hung up on him.

-1 Sirius: Don't harass your customers!

The next morning I went online to straighten everything out. Online, it showed my service was still active, and that I indeed owed money for my week and a half of service.

-1 Sirius: Customer service rep lied about canceling my service!

On the website, it says you cannot cancel online, but need to call between certain hours.

-1 Sirius: Making the customer jump through hoops to cancel

I called customer service later in the day and spoke with a very pleasant young woman named Sarah, who helped me through the cancellation process. She asked me why I wanted to cancel, and I told her about the customer service reps. She was very understanding, and even took off the pro-rated $4.95 that I owed for my partial month of service (that was only a third of what the other customer reps had said I owed, btw), in hopes that I might join back up with Sirius/XM in the future.

At the end of the conversation, I honestly considered it. After all, there was some good programming on Sirius, but I needed a few days to think about it.

-1 Sirius: Lying about how much the customer owes

+1 Sirius: Good customer service when you call to cancel

+1 Sirius: Taking off the $5 from my bill

I felt good about my Sirius XM experience until just a few hours later when yet another hard-to-hear call from yet another foreign person about my service. This time, after going back and forth about not being able to hear the person on the other end of the line, I finally understood that she was calling me to inform me that my service was canceled!

Absolute madness.

-1 Sirius: Harassing your customers after they leave

That brings our tally to -4 against Sirius/XM, and a great lesson in how not to treat your customers.

You don't need to worry about GPS

2014-01-07T11:03:21-05:00

GPS is virtually synonymous with navigation, but GPS exists at the whim of the US government, so shouldn't I be concerned? The answer is that I am somewhat concerned, but that I also do my part in trying to reduce our dependence on that system.

I'll venture to bet that everyone reading this knows what GPS is, but I suspect fewer people realize that GPS is entirely owned and controlled by the US Department of Defense. In other words, the millions of GPS enabled devices work because the US government allows them to work, and if the US decided to shut the system down, even temporarily, it could mean disaster. Am I worried? Not really.

Disabling GPS Would be a Financial Disaster

The GPS system is used worldwide, in everything from airplanes to cell phones. If that system was degraded sufficiently, or shut down even temporarily, it would cause a global financial disaster.

Even if the US were to only select certain satellites and to degrade the signal over certain regions, there would likely be unforeseen consequences which would eventually effect the US financially.

In addition to any immediate consequences, there would also be long term issues raised around US reliance, which is a topic of increasing concern internationally, especially after recent NSA revelations.

GPS is Not the Only Game in Town

Even if the US were to shut down its system, the US's GPS system is not the only satellite navigation system. Russia's GLONASS system, China's Beidou system, and Europe's upcoming Galileo system are all offering services which are similar in functionality and quality to GPS, and with all in one chips coming out, there will be even less incentive to degrade any one system's service.

Ground Based Location Systems Exist

In addition to satellite navigation systems, we can, and often do use ground based navigation systems. Airplanes use a variety of land-based augmentation systems such as WAAS and cell phones can determine their location by use of cell tower triangulation.

In addition to these methods, by looking at other signals, such as Wifi access points, one can lookup their location. Both Google and Apple use a combination of cell tower data and wifi access points in augmenting their GPS systems.

The Mozilla Location Service

There have been a number of cell and wifi collection services. All of them have so far been either technically lacking or proprietary in nature. That's hopefully changing with the Mozilla Location Service, a new Mozilla project to provide an alternative to the popular proprietary services, including source code to both the server and client and, once the privacy issues are sorted out, a downloadable database of locations.

My personal contribution to the project is that I run their data collection client, MozStumber on my Android phone, and I encourage others to do the same.

A Free GPS Service?

Would it be better if we had our own navigation system owned by the community, used as a common resource in the same way that Free/Open Source software, and the OpenStreetMap project serve the community. Of course it would, but do to that, we'd need millions of dollars in development, construction, launch and maintenance of the satellites needed for the effort. Even with advanced in amateur rocketry and satellites, I don't think that this project will be feasible in the near future, but it's also not necessary with so many existing alternatives.

So don't worry about knowing where you are, you're covered.

The World Cares About OpenStreetMap

2014-01-05T23:26:45-05:00

[UPDATE] I will stop updating this, but after four translations, syndication by The Guardian, and a video interview on Slashdot... I'm speechless.

Wow, what a response! My last post, received over 120,000 unique visitors, over 500 tweets, was #1 on Hacker News, was #2 on /r/technology on Reddit, and a mention by the CEO of DuckDuckGo, and more.

We also saw the number of new user accounts per day triple from ~33 accounts per day to just over 120 accounts per day, which doesn't include the buildup and the cool-down, which also exceeded our signup rate.

I also received several nice pieces of email from folks who read the post and either had followup questions, or just wanted to thank me for writing it.

I'm writing this post to express my gratitude to everyone.

With any group, it's easy to lose perspective, or get lost in an echo chamber of your peers. I wasn't sure who would care to read about a project they likely never used, but based on the response, I think the answer is that there are a lot of people. That's really validating and helps me get through the tougher times in the project.

For new folks, I won't try to re-hash the many new user introduction guides out on the web, but in future posts, I'll try to talk more about the internals of the project- how things work behind the scenes. For regular editors, I'll be talking about some exciting work that I'm doing to make our project better.

Stay tuned because it's going to be a really fun ride.

Thanks again for the support!

Update 144k unique views, >700 tweets, >100 Google+ posts, and nearly 350 Facebook posts.

Oh, and GIZMODO SYNDICATED IT!

Why the world needs OpenStreetMap

2014-01-04T20:46:25-05:00

Every time I tell someone about OpenStreetMap, they inevitably ask "Why not use Google Maps?". From a practical standpoint, it's a reasonable question, but ultimately this is not just a matter of practicality, but of what kind of society we want to live in. I discussed this topic in a 2008 talk on OpenStreetMap I gave at the first MappingDC meeting. Here are many of same concepts, but expanded.

In the 1800s, people were struggling with time, not how much of it they had, but what time it was. Clocks existed, but every town had its own time, "Local time", which was synchronized by town clocks, or more often than not, church bells. Railway time, then eventually Greenwich Mean Time eventually supplanted all local time, and most people today don't think about time as anything but universal. This was accomplished in the US by adoption first of the railroads, and then by universities and large businesses.

The modern day time dilemma is geography, and everyone is looking to be the definitive source. Google spends $1 billion annually maintaining their maps, and that does not include the 1.5 billion Google spent buying Waze. Google is far from the only company trying to own everywhere, as Nokia purchased Navtek and TomTom and Tele Atlas try to merge. All of these companies want to become the definitive source of what's on the ground.

That's because what's on the ground has become big business. With GPSes in every car, and a smartphone in every pocket, the market for telling you where you are and where to go has become fierce.

With all these companies, why do we need a project like OpenStreetMap? The answer is simply that as a society, no one company should have a monopoly on place, just as no one company had a monopoly on time in the 1800s. Place is a shared resource, and when you give all that power to a single entity, you are giving them the power not only to tell you about your location, but to shape it. In summary, there are three concerns- who decides what gets shown on the map, who decides where you are and where you should go, and personal privacy.

Who decides what gets displayed on a Google Map? The answer is, of course, that Google does. I heard this concern in a meeting with a local government in 2009- they were concerned about using Google Maps on their website because Google makes choices about which businesses to display. They were right to be concerned about this issue since a government needs to remain impartial and by outsourcing their maps, they hand the control over to a third party.

It seems inevitable that Google will monetize geographic searches, with either premium results, or priority ordering, if they haven't done so already (ie is it a coincidence than when I search for "breakfast" near my home, the first result is "SUBWAY® Restaurants").

Of course Google is not the only map provider, they're just one example. The point is that when you use any map provider, you are handing them the controls- letting them determine what features get emphasized, or what features may not be displayed at all.

The second concern is about location. Who defines where a neighborhood is, or whether or not you should go. This issue was brought up by the ACLU, where a map provider was providing routing (driving/biking/walking instructions) and used what they determined to be safe or dangerous neighborhoods as part of their algorithm. This begs the question of who determines what makes a neighborhood safe or not, or whether safe is merely a codeword for something more sinister.

Right now, Flickr collects neighborhood information based on photographs which they expose through an API. They use this information to suggest tags for your photograph, but it would be possible to use neighborhood boundaries in a more subtle way in order to effect anything from traffic patterns to real estate prices, because when a map provider becomes large enough, they become the source of "truth".

Lastly, these map providers have an incentive to collect information about you in ways that you may not agree with. Both Google and Apple collect your location information when you use their services. They can use this information to improve their map accuracy, but Google has already announced that is going to use this information to track the correlation between searches and where you go. With 500 million Android phones, this is an enormous amount of information collected on the individual level about people's habits whether they're taking a casual stroll, commuting to work, going to their doctor, or maybe attending a protest. Certainly we can't ignore the societal implication of so much data in the hands of a single entity, no matter how benevolent they claim to be. Companies like Foursquare use gamification to overlay what is essentially a large scale data collection process, and even Google has gotten into the game of gamification with Ingress, a game which overlays an artificial world onto this one and encourages users to collect routing data and photo mapping as part of effort to either fight off, or encourage, an alien invasion.

Now that we have identified the problems, we can examine how OpenStreetMap solves each of them.

In terms of map content, OpenStreetMap is both neutral and transparent. OpenStreetMap is a wiki-like map that anyone in the world can edit. If a store is missing from the map, it can be added in, by a store owner or even a customer. In terms of display (rendering), each person or company who creates a map is free to render it how they like, but the main map on OpenStreetMap.org uses FLOSS (Free/Libre Open Source Software) rendering software and a liberally licensed stylesheet which anyone can build on. In other words, someone who cares can always create their own maps based on the same data.

Similarly, while the most popular routers for OpenStreetMap are FLOSS, even if a company chooses another software stack, a user is always free to use their own routing software, and it would be easy to compare routing results based on the same data to find anomalies.

And lastly, with OpenStreetMap data, a user is free to download some, or all of the map offline. This means that it's possible to use OpenStreetMap data to navigate without giving your location away to anyone at all.

OpenStreetMap respects communities and respects people. If you're not already contributing to OSM, consider helping out. If you're already a contributor- Thank You.

Update Wow, what a response!

Learning Math Earlier With Computers

2014-01-04T00:15:24-05:00

Writing the previous post got me thinking about the nature of how programmers use programs and how it relates to teaching in general.

A couple of years ago, I wrote a little prototype address book that accounted for my misspellings. The idea was that if someone had a name like Karalyne, that even if I spelled it "Caroline", it would find it.

To do so, I used the Soundex algorithm in Python:

>>> import fuzzy
>>> soundex = fuzzy.Soundex(5)
>>> soundex('Caroline')
'C6450'
>>> soundex('Karalyn')
'K6450'

Then I used the Levenshtein algorithm to determine the best match, and as the user typed, the matches would become better.

When I wrote the program, and even today, I have no understanding of exactly how Soundex works. I know that it transforms the string into a set of values which correspond to how it sounds, but I don't need to understand its implementation to use it.

That's the way most libraries work- one doesn't need to understand them in order to use them.

In this TED talk, Conrad Wolfram argues that this is the same way that we should be teaching math to children, with an emphasis on usage first, and understanding afterwards.

His examples include geometry, but the same arguments could be made about statistical analysis.

Most people working with a dataset will need to know what questions to ask- determining the type of data and knowing what questions to ask is the most important thing. Figuring out whether or not the data fits a certain model will help you determine the type of analysis that needs to be done next.

If we taught statistics this way, with an emphasis on the analysis, rather than the process, we could teach statistics not in college, as Arthur Benjamin suggests, but as earlier, even as early as Algebra. The concepts in a subject like statistics are concrete, they're tangible, and it would be possible for students to be as comfortable using them even before they understand them, just as I'm comfortable using Soundex, even when I have only the most rudimentary understanding of how it works.

Python for Adults

2014-01-03T11:42:00-05:00

My girlfriend has always wanted to program. She even took C++ in high school and college, but never quite "Got it", leaving her with a bad taste for computers and programming. Still, when I lent her my copy of The Little Schemer, she took to it pretty easily, so I knew she had the potential to be able to program, despite her past difficulties.

So then the question becomes what language to teach her. While Scheme would be a nice choice in theory, despite more than three decades as a teaching language, it's still not popular in the mainstream. While this wouldn't be important in a classroom setting, adults have a different set of needs for programming. Notably, they need to be able to apply their skills quickly to something in their life.

That's why Python fits the bill so well. First off, its simple. It's syntax is absolutely minimalist, as are its features. I call Python by "80% Language", in that 80% of all tasks I need to do, I can do with Python. It doesn't have Ruby Blocks, or Interfaces, or Lisp Macros, Python is arguably the best general purpose language out there.

The key to Python's success is three part. The first of those is design. Python's roots as a teaching or prototyping language meant that there is an emphasis on simplicity and elegance. It's very light on syntax, it supports Object Oriented programming, but does not demand it, and operations that are performed often, like array indexing, hash table lookups and mapping have special syntax to make them more convenient.

Python also stays firmly in the general purpose language camp. R has become increasingly popular for statistics work in the last five years, and while R does have extensions, it is designed for data analysis. Perl started off its life as a text processing and scripting language, and while it's grown out of that role, it still has those concepts embeddeded within it. Python changes slowly over time and borrows from other languages while staying true to itself. For example, Python recently adopted Asyncronous IO in the core, but does not require that any program use it.

The second strength Python has are its libraries. While Python has fewer libraries than Perl, it offers more simple, practical interfaces to more systems. For example, I mentioned R earlier, and Python programmers can interface directly with R through RPy, while still being able to use simple web accessing frameworks like Requests, and powerful phonetic algorithms such as those found in Fuzzy. In fact, whole Python distributions have been created around nothing but packaging up the best libraries for a particular task, such as SciPy, and Biopython.

The third, and perhaps most important strength of Python is its user community. I'm sure this will be the most controversial part of this post, but I've found the Python community has bar-none the most supportive users. This is not by accident, but part of Python's legacy and current commitment to inclusion. Python came from a teaching language background, and documentation was, and continues to be part of that legacy. Python is used as a teaching language in High Schools as well as MIT.

In addition, the Python Foundation focuses a lot of attention and energy into community diversity through its Diversity Statement, as well as commitment to bring women into the community, both through their own local communities (PyLadies) but also focusing on bringing that diversity into mainline events such as PyCon. The net effect is that Python is not only welcoming to women, but has a general welcoming atmosphere to people of virtually any background.

If someone is considering Python, especially if they have a scientific background, I suggest watching Science and Python, and jumping in with Learn Python the Hard Way followed with New Coder, which is described in depth at this talk by Lynn Root.

Compressor Repair

2012-06-09T13:24:00-04:00

My friend Chad of the band Emperor X has put out a video for their new song "Compressor Repair".

I'm a huge fan, so indulge me and check it out.

Reintroductions

2012-06-09T08:30:00-04:00

Wow, I must have really touched on something in my last post (Observations at a Python workshop), it's received over 20k unique visitors in the last twenty four hours, which is a lot from a blog which until yesterday, didn't exist.

For those people who don't know me, I thought I'd give some introductions.

My name is Serge Wroclawski, I'm a long standing member of the Free Software community. I spent 12 years of my life as a Unix system administrator, mostly working for academic government institutions such as NIH and NASA.

I've quit my job to work on my passion, which is to help the various communities involved in software, culture, education- basically to make works available to anyone and everyone who wants it, without restrictions.

To that end, I'm in the process of founding a new non-profit dedicated to Free Culture and making it easier for artists to add their work into Free Culture. If anyone would be interested in helping in this effort, please feel free to contact me.

I'm also moving to New York City later this summer and would love to meet some new people there, especially those in either the arts or technical community, so if you're in/around NYC drop me a line!

Thanks again everyone for reading/subscribing. I hope to have lots of posts in the days to come; my personal goal is one not less than every two weeks.

And if anyone is curious, my old blog is still around for now, though I am not updating it, and it's full of comment spam.

Observations from a Python Workshop

2012-06-07T23:09:00-04:00

I heard about a Python tutorial that was going on this weekend, and with well over a decade of Python experience, I decided to volunteer to help students. I hadn't realized it was a women's workshop put on by PyLadies, but I'm glad I went because I came out of the event with a number of observations that extend out to the Free Software community as a whole.

It's no secret that our community isn't great at encouraging women contributors. Various surveys show that between 1.5 and 5% of Free Software contributors are women. That is low even compared to the 12-25% of women in computer science. And while we all cringe at incidents like the Golden Gate Ruby Conference fiasco, and the Squoot Jam, these incidents alone don't explain the lack of participation by women in our community.

The Audience Is Out There

The class was well attended. Seventeen women and one man came to learn how to program in Python. Considering that Python materials are readily available in books, on the web and in videos, this heavy attendance shows that there are people who want to learn and that they will if they're presented with the opportunity.

Motivation

In one of the classic programming texts (I forget which one), the author says that very few people who learn to program should be programmers, but that he'd like to see more programming done by those of other professions. The workshop would have made him proud, as every student I talked to shared the sentiment with me that they hoped to learn Python in order to do their job more effectively, by learning to collect, sort or process data in new and more efficient ways.

The Students Care

During the first half of the class, the students were engaged, and excited. They asked questions, and worked with the instructor, and with each other, to help fill in the gaps. Sometimes they'd ask a volunteer for help, but largely the questions I received were grammatical rather than conceptual.

The Unsure Student

By and large the students seemed comfortable with the material and engaged with the instructor, but there were exceptions.

One of the students who sat next to me was a young woman who was far ahead of the class. She was easily an hour ahead of the other students in the progress she'd made with the final project when she asked me for help.

Looking at her screen, she'd made significant progress towards finishing the project, but when she asked me a question she giggled and told me she had no idea what the project was about, or what to do.

Why?

I still don't know the reason. Studies have shown that especially bright women have this reaction to difficult material and that women learning computer science do better in an single sex classroom (though those studies have critics). But I can say that I personally found this experience surprising, and I wondered afterwards why this extremely competent woman in her 20s would react this way after clearly mastering the material.

The Presence of a "Helpful" Jerk

I mentioned earlier that the first half of the class had a lot of student engagement, with questions and joking.

After lunch, a new volunteer joined the classroom, someone I'll call Dave.

Dave was really excited by the class, and, like me and the other volunteers, had a lot of experience with Python.

Dave expressed his enthusiasm for the material by asking a lot of very technical questions to the instructor. Largely, these questions seemed designed to show off his own knowledge, or to challenge the instructor. In other words, Dave was the jerk at the conference.

And Dave's behavior had a noticable effect on the workshop. Students tuned out, became distracted, asked less questions, and had a lot more difficulty following the material.

While the students tuned out, I felt myself getting angry. I saw Dave's behavior as a challenge, a call to action. And it was at that moment that the gender difference became most apparent to me.

To me, and I suspect most men, we'd see Dave's behavior as a call to arms. Dave was raising the stakes and even though his behavior was clearly rude, it also demonstrated that he had a mastery over the material.

But to this classroom, Dave's behavior not only ground the timing of the class down to a halt, it also seemed to have an effect on the students, who had until then been very involved in the workshop.

Lessons Learned

I'd never really believed in single sex education, and I'd really never believed that the gender difference in computer science education was due to lack of interest. Just as I've never heard an outcry about how few male teachers exist, I never understood the outcry about the lack of women programmers or sys-admins.

But my experience last weekend changed my view. One of the students from the workshop came to the DC Python meetup, and I saw that women only groups are not creating a more insular community of women, but rather offering these women a more comfortable entrance into our general community, and so I say more LinuxChix, more PyLadies, and more Ladies Learning Code.

The jerk factor also reminded me of why we in the Free Software community need to be taking issues of civility seriously, and not letting the idea of "free speech" get in the way of maintaining a safe place for discussion. I've seen groups fail in both ways, by allowing jerks to run the roost, and with draconian and arbitrary moderation. I can't recommend Art of Community enough.