Campaign +1. Collecting 1000000 signatures

​It took us 4 months to collect the 300,000 signatures needed to nominate a candidate for the 2018 presidential election. Our new goal is one million and we are launching the "+1" campaign.

According to the laws of the Russian Federation, in order to be registered as a presidential candidate, the applicant must submit 300,000 confirmed signatures to the Central Election Commission, which must be collected in a very short term. Only 40 days are allotted for the collection, and in our case even less, because these 40 days fall on the New Year holidays. In fact, this is another artificial obstacle created by the authorities. In addition, signatures must be collected in 40 regions of the country - no more than 7,500 from each subject of the Russian Federation.

Photo: Evgeny Feldman

In order to submit the necessary 300,000 to the CEC on time, we must know that in all regions where our headquarters will be opened, we can count on 7,500 signatures. And that they will be collected on time and in full compliance with the procedure.

Those candidates whom the Kremlin wants to see in the elections, and cut paper will be taken as signatures, we have seen this more than once. Our signatures will be viewed under a microscope

Alexey Navalny

Photo: Evgeny Feldman

Most likely, the CEC will study the signatures submitted by Navalny's headquarters especially carefully - therefore, we must insure ourselves as much as possible and collect much more than is prescribed by the protocol. We collect not only email addresses, but also phone numbers and brief profiles of everyone who is ready to support the nomination of Alexei Navalny. This is done so that we know exactly in which regions there are our supporters and how many there are.

There are a lot of people around you who want to support an independent opposition candidate. At least for reasons of competition in the elections. Even zaputintsy support competition, they want to see independent candidates on the ballots

Alexey Navalny

Our next goal is one million signatures, which will guarantee us the required 300,000 at the end of the year. To do this, we are launching a “+1” campaign and calling on everyone who has already signed on the site to convince at least one more person to register. And better than two or five - and then the required million will be reached by the summer.

Now we really need your help. Please take 10-15 minutes of your time and encourage one of your friends, relatives or work colleagues to sign.

Photo: Evgeny Feldman

Write a letter to several friends or a message on a social network:
"Hey. I put my signature for the nomination of Navalny as a candidate. Could you put it in too? It will be right"

Alexey Navalny

The 385,531 people who have already signed up are a formidable force. Let's make the most of it. Join the "+1" campaign.

The signature sheet is the main document in our system. The first thing you want to do when working with a large collection of objects is to give them a unique ID to associate each object with a record in the database. But the form of the signature list is very strictly prescribed in the law, any violation of it is a reason to reject all the candidate's signatures in general. On the sheet, which is submitted to the election commission, no extra marks and symbols are allowed.

When collecting signatures in Novosibirsk, we placed each sheet in a multifora (transparent “file”), on which the sheet ID and all service notes were written with a marker. This worked for four thousand sheets, but it won't work for hundreds of thousands. This time, we considered the use of multifors to be an unreliable and inconvenient solution.

Lawyers invented a method that allowed us to identify each sheet and not violate the form of the signature sheet. The law does not say anything about the physical size of the signature sheet. This allowed us to design the sheet in such a way that identification codes are applied to its upper part, and are simply cut off before submission to the election commission.

The sheet code consists of 6 characters. You can use Latin numbers and letters that have graphic analogues in Cyrillic (you can write in forms in any layout). Separators have been added for convenience: 91−X7−BA.

The same identifier is printed as a QR code for automatic recognition at various stages of work. QR codes won over all other types of barcodes in terms of reliability and speed of recognition.

The life of headquarters is full of difficulties, so QR codes were thoroughly tested in various stressful situations for sheets…

... and decided that three codes would be enough to process any live sheet.

Lawyers and designers have worked hard to make the layout comply with both the law and common sense. Separately tested the number of signatures on the sheet. Few signatures - too many sheets, a lot of unnecessary writing (data of the collector and trustee), more errors in the certification inscription. Many signatures - it is inconvenient to enter the voter's data, more errors in signature lines. After experimenting with prototypes, they settled on five signatures.

Each sheet (more precisely, the sheet ID) is created in the database, after which it can be printed on A4 paper. But you can’t just take and print a sheet on the nearest printer. According to the law, the production of signature sheets must be paid from the candidate's electoral account. They are usually made by an external contractor. Therefore, we made the technical side as friendly and flexible as possible. Sheets are either printed directly from the browser, or pre-saved in a multi-page PDF file that can be transferred to the contractor in any convenient way.

Sych: preparing for the collection of signatures

The collection of physical signatures in signature lists can only be started after the nomination of a candidate and the opening of a special electoral account. The law allocates very little time for this. It was important for us to do as many operations as possible in advance in order to debug all processes and, after the official announcement of the elections, speed up the work as much as possible. For a preliminary check of the data of our supporters, for training headquarters and testing the collection mechanics, we launched a verification procedure.

Verification is a beta version of the collection of signatures: in real headquarters, with the same equipment, with the same strict document checks, but without entering a signature on a paper sheet. To work with the data of verified people, the Sych application was developed.

The composition of the Owl

Backend with RESTful API: Python 3.6, aiohttp, aiohttp_admin, SQLAlchemy.
Databases: PostgreSQL, Redis.
Notification daemon.
Passport number recognition daemon.
Daemon for building analytics.
Passport verification service by its number.
Boxed version of Cladr-API for working with addresses (PHP 5.6 + MongoDB).

We decided to make a separate backend for Sych with a RESTful API, because we planned to integrate it with several services, including the Navalny 20!8 website. We used a separate PostgreSQL database as storage and Redis for caching. To manage users, the aiohttp_admin library came up, which we modified to fit our needs.

The internal operator interface is a step-by-step form of passport scanning and filling in personal data. Due to the large number of possible states, this form was written in React.

Interaction with the site "Navalny 20! 8" was carried out through the API, which is protected by a token and is available only via local network between virtual machines.

Registration for verification

In order to evenly distribute the load on the headquarters, we came up with a record for verification. After registering on the site, a person got access to the recording interface, where he chose a convenient headquarters and time.

To control workload, manage records and schedules, we have developed a separate interface available to the regional manager and headquarters coordinator:

If the headquarters has an emergency, the coordinator can massively cancel future entries for verification. However, he cannot do this on his own - he must request a cancellation confirmation code from the regional manager. We had to repeatedly use this option.

Notifications

In Sych, a branched notification system was implemented. The signer should have received notifications by mail when he signed up for verification, missed the registration, a week after the cancellation of the registration, after successful verification, after the headquarters canceled the registration, and in several other cases.

SMS notifications were sent to remind the recording three hours in advance and to inform that the headquarters canceled the recording. The notification queue was made according to the same principle as on the Navalny 20!8 website: tables in the database with messages that were sent in groups through mail and SMS gateways.

Passport data recognition

To evaluate the work of operators and determine the percentage of data entry errors, I wanted to have additional scan recognition. Reliable automatic recognition was impossible due to the variability of passports, so two options were considered: send scans to Yandex.Toloka to be recognized by its users, or take a group of volunteers who would do this in the office. But the issue of personal data security stopped both options, and we left automatic recognition only for the passport number.

Analytics Sych

During verification, we not only clarified and checked our base of supporters, but also tested the work of headquarters, infrastructure, equipment, and the mechanics of collecting signatures. To monitor the process and correct it, we made a simple analytics.

Since the headquarters has three levels of process management - headquarters coordinators (responsible for the work of one headquarters), regional managers (monitors a group of headquarters in several regions) and federal headquarters management (monitors everything and everyone), the system grouped data in different ways. for each category of users.

We showed most of the details to the headquarters coordinator. He saw the statistics of all operators and the dynamics of key indicators and could make management decisions based on them: appoint more or fewer operators, increase alerts, change work schedules on weekends, fire or re-educate employees who often make mistakes, etc.

We saved the regional manager from unnecessary details, and on the first screen he saw only the most important things for his group of headquarters: key indicators, ratings and problematic headquarters (indicated by alarming red). We classified “problematic” headquarters with indicators N% below the average, chronically underloaded (they needed additional notification) and overloaded in terms of the number of appointments (which meant that not all people could sign up and the number of operators needed to be increased).

To better deal with the discovered problem, the regional manager could easily look at detailed statistics for each headquarters and see all the data that is available to the coordinator.

It was important for the federal headquarters to immediately see the full picture, so we collected the key campaign metrics on one screen and made a summary table for all cities where verification is taking place. In the table, you can select the headquarters of interest to view the full set of data on it.

In total, more than 50 indicators were displayed in the analytics. The flexibility of SQLAlchemy was enough to never switch to pure SQL and to keep the code readable. For the most labor-intensive indicators, caching was first done in Redis, but it turned out to be easier to periodically calculate them in the background and take them from the file when making requests.

Reaper-2018: a system for collecting signatures

In parallel with the verification process, a system was developed for collecting signatures. The architecture of the system used in Novosibirsk and able to work with physical objects - sheets and signatures was taken as a basis.

From the side of the backend, Reaper-2018 is the heir to the old Reaper, but it received the operator interface from the verification system. Some screens have been finalized after analyzing feedback on Owl's work. In addition, interfaces have been added for several levels of data validation and for sheet movement control.

Operator interface

In the process of obtaining a signature, the operator must scan the voter's passport, fill out the questionnaire (taking into account that the address indicated in the registration stamp may not be written in the right format at all) and enter data into the signature sheet following the instructions of the system. But first we have to check if the voter meets three key conditions:

1. At the time of the election, he must be over 18 years of age.
2. If the voter is 20 or 45 years old, he must have a new passport.
3. The passport must not be listed as invalid.

Checking the database of invalid passports is a simple operation, but it also has its own subtleties. The base is distributed by the Ministry of Internal Affairs on its website. Previously, before the elections, for some reason they turned off the ability to unload this database, so we started downloading the current version of the database daily in advance (don't forget to turn it off).

Now there are more than 110 million records in the database (series and passport numbers). For a quick search with a small amount of database and indexes, the following scheme was invented: PostgreSQL creates a table with a million records, the primary key in which is the passport number (from 0 to 999999), and the second field contains all series of invalid passports for this number. To reduce the size of the series, they were converted to binary format (two bytes each) and compressed using zlib (just wanted to). Initially, the database takes about 1 GB, excluding indexes. After processing, 260 MB is obtained along with the index. One record is checked on average for 15 ms.

0.6% of the passports of people who underwent verification were found in the database of invalid passports. This means that without such verification, we would spend 12% of the invalid signatures limit on this type of error only.

0.88% of the passports did not suit us, because the citizen is 20 or 45 years old, but he has not yet replaced the passport. And this is another 18% of the limit of invalid signatures.

The signature list contains 4 columns filled in by the operator: full name, year of birth, passport number and address of permanent registration. All this data passed through the Reaper to check and correct possible errors. For example, in the fields for the name and patronymic, the search for typos works:

For name hints, the API has a method that compares a value against a large list and returns three responses:

Everything is OK, there is such a name;
- there is a similar name (such and such);
- unknown name (rare name or serious spelling errors).

A separate story is the letter "e". There are passports that use it, but in most cases it is replaced with "e", so we display a warning if there is an "e" in any field of the passport data.

The system does not correct anything itself, it only informs. The operator and inspectors should pay attention to such cases and make the right decision.

Scanning Documents

To obtain images of documents, we use scanners of our own production, and Raspberry Pi as an operator station. This is described in detail in the second chapter.

This image is not a passport scan, but is collected in a graphics editor from random data.

The image is obtained on the client side from the HTML 5 Canvas API and sent to the server as a base64 string containing a JPEG. From the front-end point of view, scanners can work in two modes: USB webcam and video stream streaming from a computer on a local subnet. Owl only works with USB cameras, while Reaper-2018 allows you to switch between modes. The operator chooses which scanner to use.

There was a small problem with choosing the video stream of neighboring computers: tables and scanners can be moved, and operators can change seats. We don't know which scanner will be next to the operator next. I had to go through the headquarters subnet and give the operator the opportunity to select any of the live scanners. But it turned out that the scanner's video broadcast server, although it sets the correct CORS headers (Access-Control-Allow-Origin: *), does not respond to OPTIONS requests. The browser disallowed ajax requests to neighboring hosts, which prevented us from using regular jQuery.ajax() for searches. JSONP requests didn't help either, as they couldn't be canceled programmatically, and several dozen pending requests completely blocked the page. Pictures helped solve the problem. We added tags to the DOM and assigned the src of the video stream to them. If the picture changed sizes in accordance with the size of the stream, then the stream was considered alive and was shown to the operator.

Displaying the video stream in the browser significantly loads the modest Raspberry Pi processors, so we had to make a “screensaver”: after 5 minutes of inactivity, the browser pauses the broadcast.

It is important for us to choose up-to-date information about the place of registration. There can be 6 stamps on the front of the passport, but only one is needed. The interface offers to select it using the arrows on the keyboard or by clicking on the desired stamp on the preview.

There may or may not be registration. Such voters are recorded in a separate signature sheet with an empty region and address, and registration scanning is skipped.

Address processing

The most difficult part in filling out the signature sheet is the voter's address. More than half of the errors that invalidate a signature are related to the address.

There is a long list of legal requirements for the registration address. For example:

This must be an address according to the FIAS database (federal information address system);
- for renamed streets, new names must be indicated, even if the passport had the old one;
- the law establishes a certain format for the hierarchy of address objects that need to be recorded (for example, you cannot specify an urban area).

These are only the basic points, but there are also many little things, the list of which was replenished with each interaction with the election commission. Failure to comply with even minor requirements is a reason for the election commission not to accept the signature.

At the collection of signatures in Novosibirsk, about 3.5% of signatures were declared invalid due to complaints about the "address" field. And this is 70% of the limit set for signatures for the nomination of a presidential candidate.

In order to fulfill all the requirements, we are forced to run each address through a computer in order to form the correct format and indicate to the collector, to the nearest character, what he should enter in the signature list.

We try, whenever possible, not to use the API third party services, so as not to give away data about our users and not to end up in a situation where the API is suddenly turned off at the most crucial moment. Working with addresses is a critical function for collecting signatures, so we had to create our own API for the FIAS database.

The FIAS database does not yet have sufficiently high-quality and complete information about houses and apartments, so we stopped at street level. In this form, the database with all additional constructions weighs about 2 GB and lives quite comfortably in the form of PostgreSQL. For the import, modified scripts from the fias2pgsql repository were used.

For a universal all-Russian form for entering an address, you cannot simply make the fields "city", "street", "house", because there are many different address formats and types of address objects. A well-known example of an unusual format is Zelenograd, which has houses without a street name. But, believe me, on a national scale this is a rather trivial case.

After a series of experiments, we settled on a form with three fields:

The subject of the Russian Federation - it is always there, this is the most understandable field;
- FIAS address - a field with auto-completion for the addresses of the given region within FIAS;
- house / building / apartment - a line where the data is rewritten exactly in accordance with the permanent registration stamp.

Lawyers compiled an address conversion table, with the help of which we brought FIAS addresses into a format that complied with the electoral legislation. Most often it was necessary to exclude one of the elements of the address. Some addresses were completely excluded (garage cooperatives, yard areas and other similar objects). The IT department received a table of rules, and the legal department received 10 examples for each of the 44 address types.

After several such iterations, the database was ready to go.

The technical part of the task was to organize a convenient and fast search with auto-completion, which will withstand a load of 1 million requests per day. Sphinx was used as a search engine. The request is cleared of extra characters and passed to Sphinx, and it returns the full addresses of objects, ranking them according to the specified rules.

Sphinx indexes the address field written in XML format. This storage format turned out to be convenient in that all metadata can be hidden in XML attributes that Sphinx does not use for searching, but keeps in memory and returns in the results without additional access to the database. Somewhere on the frontend, these attributes are used to form a pretty address bar.

The solution turned out to be convenient and fast. One request to the suggest API is completed in 15–20 ms, the backend quietly processes 300 simultaneous connections on a not the most powerful virtual machine.

Filling out the signature sheet

Signatures must be entered on the sheets of that subject Russian Federation, to which the address of the permanent registration of a citizen belongs (or to special lists without a region, if there is no registration). The reaper informs the operator which region's sheet to take, and does not allow the signature to be signed on another region's sheet.

Imagine that you want to solve such a problem without a computer, collecting signatures at the station, where there will be many people from different regions and there will be no filing cabinet with blank sheets sorted by region. In about a third of passports, the registration stamp does not contain the name of the region, and random passers-by do not know the rules of the game and can easily mix something up. This looks like a source of a lot of errors, which is unacceptable under the legal limit of 5%.

Filling out the signature sheet is a complex and responsible procedure. The sheet has lines of signatures, an acknowledgment inscription of the collector and the signature of a trusted person. All these blocks must be completed in accordance with strict formal requirements. At each stage of filling, errors are possible that can invalidate the entire sheet or part of the signatures.

We have developed operator scenarios that reduce the likelihood of common errors. The certification inscriptions of the sheets of the "home" region (about 80% of the signatures will be from the region in which the headquarters is located) are filled in by the collector in advance, in a calm atmosphere. For all blocks of the sheet, the Reaper shows exactly how they should be filled.

The filling interface imitates a real signature sheet, which in this moment lies on the table in front of the operator. Busy lines, columns to be filled in, sheet number, large data to be entered are shown.

For a filled line, the operator must indicate its status (it is not always possible to fill in the line successfully the first time). Each correction and deletion must have a note from the assembler on the sheet and a corresponding status in the database.

After filling out the entire sheet, the date and signature of the collector are affixed on it. The sheet is submitted for review.

Verification of signatures, work with sheets at the headquarters

At the end of each working day, all sheets with signatures are sent for verification, which takes place late in the evening or at night (we have small headquarters, there is simply nowhere to conduct all processes in parallel). The inspector (who is also the candidate's confidant) looks through each sheet and each signature, compares it with fragments of scanned pages of the passport, and checks all significant elements against the checklist. If errors are found, this is noted in a special interface.

The certification record is checked separately. Certification errors are especially dangerous because they affect the entire sheet at once. Such errors account for the origin of approximately 9% of all invalid signatures.

Some errors can be corrected, but only the collector can make corrections to the signature lines, and he is not at the headquarters in the evening / night, so all the information necessary for correction is transmitted electronically. To understand the context, you need to see everything that happened to the line before. So there was a "chat" between the inspector, the operator and the lawyer.

All names and other data in the image are fictitious

If the errors seem fatal or there is any doubt, the sheet is sent to a lawyer. If the signatures do not contain errors or all corrections have already been made, the verifier signs the authorized person and sends the sheet to be sent to the central headquarters.

Emoticons and the Neurophysiology of Happiness

For quick and error-free selection of the status of the line being checked, we used buttons in the form of emoticons. There are deep neurophysiological reasons for this. There are ancient low-level mechanisms in the visual system of the brain that respond to certain images. The visual system reacts most quickly to segments of straight lines of different orientations, since the lines are easily detected by the primary visual cortex. In the secondary visual cortex, simple geometric shapes are recognized (this needs to be learned) and the scheme of the face. Moreover, not just a face is recognized, but the main facial expressions. That is, emoticons. Like the recognition of straight lines, this is an innate ability. Thanks to such a low-level system, emoticons are recognized much faster and more accurately than text.

Icons in the form of emoticons well correspond to the meaning of the statuses that the verifier can assign to signatures: “good”, “have problems”, “bad”. There were some hesitations with the “show the lawyer” emoticon, but we got over it.

There is also an opinion that emoticons humanize the interface and thus slightly improve the life of the operator. This is important because operators had to spend long hours working with our system and not lose their vigilance.

Sending sheets

Finished sheets are sent to the central headquarters every day. There may be many sheets, several hundred. We want to know exactly which sheets are ready and left the headquarters, but manually registering them is time-consuming and unreliable. A mobile application has been written to account for sent sheets.

It has a mode that allows you to quickly scan the codes of hundreds of sheets and reports if a sheet is being sent by mistake, when it has not yet gone through all the stages of processing at the headquarters. It takes 1-2 seconds to scan one sheet.

After scanning, the sheets are packed and sent to Moscow.

Form details

All passport details are entered and displayed in Source Code Pro Regular monospace font. In it, zero is easy to distinguish from the letter "O", and the characters are quite similar to those commonly used in modern passports.

All forms are designed so that you can switch between fields and main buttons with a tab. The input focus is in the desired field not only when the page is loaded, but also after the error message is closed. Modal dialogs capture focus so that switching occurs only between their controls.

All buttons, when pressed, something continuous happens, show this in all their appearance. The input fields are disabled while the data is being sent. In case of errors, detailed explanations appear.

Logistics and physical storage of sheets

Shifting pieces of paper is one of the activities in which mankind has achieved incredible success. It would seem that you can go to a stationery store, buy a set for collecting signatures "Federal" and not think about the details. But there is a problem: all office solutions are too expensive. We cannot supply every headquarters with document scanners for several tens of thousands of rubles and cabinets with hanging folders for a hundred thousand, so at each stage we had to invent and create something from improvised materials.

Some facts about the physics of the process

We need to hand over 315 thousand signatures. To do this, taking into account regional quotas and a margin for various errors, it is necessary to collect and process about 1 million signatures. Each sheet can have a maximum of five signatures, but in reality it will be somewhere around 3-4. This gives us, roughly speaking, 300 thousand sheets.

A sheet of A4 paper has an area of ​​1/16 m².
The density of ordinary office paper is 80 g/m², each sheet weighs 5 g.
The height of a pack of 500 sheets is 4.5 cm for blank sheets, more than 6 cm for filled ones.

It turns out that all the collected sheets will weigh 1.5 tons, and folded into one bundle, they will be about 36 meters in height.

How to store all this?

Signature lists are printed, filled in with signatures, checked, certified and sent daily to the central headquarters. One headquarters sends several hundred sheets a day, so at this stage there should be no problem.

The most interesting begins at the central headquarters. There you need to organize a storage system that will make it easy to receive sheets from regional headquarters and work with them until the end of the collection. After the collection is completed, the sheets should be grouped by regions and stapled into folders for the election commission.

We cannot simply stack sheets in endless stacks, since lawyers may at any time want to withdraw some of the sheets from a certain sample. You need to know exactly where each sheet is, be able to quickly get it and return it back.

For quick access, a system for indexing the physical database of sheets was invented. The index consists of several levels: headquarters (box), box, folder. The address of the folder in the archive looks like this: 77−1−15. Inside each folder are 25 sheets (in random order).

In the upper left picture - a box of 500 signature sheets in paper folders.
On the right picture - a box for 2000 sheets in hanging folders.

Getting and sorting sheets

All sheets coming from the regions are scanned by an automatic two-sided scanner (it was already in the office, so we didn’t have to assemble it ourselves from LEGO and Arduino). This device can upload the result to the server via SFTP. There, the scans are run through a python script that looks for QR codes in standard places, recognizes them, and binds the scans to a common database. The script reliably processes even crumpled sheets.

After scanning, the sheets are sorted. Each sheet is scanned using a mobile application (sort mode). It finds the sheet in the system, changes the status to "arrived at the central headquarters" and shows the coordinates of the folder in which the sheet should be placed. The operator confirms that he has placed the sheet in the specified folder (closes the transaction).

Sheets of the same region are placed sequentially in the folder as long as there is space in it, so the whole process is very fast.

Backend

Reaper 2018 is built on Django with a standard templating engine and ORM. PostgreSQL is used as the database. The service parts of the system - FIAS, passport verification, work with pre-registration data - are placed in separate modules (django app) with their own databases.

The physical world of signatures is represented as several classes of objects: a signature sheet, a line in a sheet, a signature. Objects of these classes have attributes that reflect the state of the object in the real world. For state management, the “finite state machine” template (state machine, finite state machine) and the django-fsm library were used. All transitions between states are written in the form of FSM transactions, within which the necessary checks and additional actions are carried out with the object.

The state diagram looks something like this:

The position of a sheet in space is determined by the state of the rows it contains. If there are lines that a lawyer must check, the sheet receives the status “to a lawyer”. As soon as the lawyer took the sheet and entered its code in the verification interface, the sheet receives the status "at the lawyer". Thus, we always know the exact position of all the sheets and understand their immediate fate.

Testing

The signature collection system has too many different states and transitions between them to check it manually. To automate checks, all scenarios related to the work of operators and checkers are covered by tests on the django side.

It is useless to look at a system for collecting a million signatures when there are no signatures in it. To populate the database, scripts were written that initialize the typical state of the database during the collection process so that you can look at the system filled with something similar to real data.

The collection of signatures is very limited in time, and a significant part of this time falls this time on New Year's holidays. We expected that the load on headquarters and the collection system would be uneven. It was important that the system easily coped with any realistic flow of signatures. At peak times, up to 10,000 signatures per hour were expected. For a normal website, this does not seem serious, but in our case, such an order of "visitors" can create a large load on the server. It's not just visits or registrations: obtaining each signature involves about 50 requests to the server and processing several high-resolution images.

Load testing was carried out using Locust. This is a simple tool available through PyPI. Scenarios are described as python code, much like unit tests in Django:

Tests can be run through a web interface that displays graphs of request rate, number of clients, and server response time.

The deployment of the project is organized in the same way as for the site "Navalny 20! 8".
Access to the Reaper web applications is possible only through the headquarters VPN network.

Monitoring

We use various tools to monitor the servers and applications involved in the signature collection system.

Zabbix keeps track of the status of all project virtual machines.

Elasticsearch collects nginx logs from all virtual machines, Kibana shows this in the form of graphs.

All errors from applications and from frontends fall into Sentry. Frontends are moved to a separate “organization” so as not to spoil the statistics on backend errors. A handy feature, but getting Sentry to work under our load was quite difficult.

goose

This is a functional monitoring, somewhat similar to uptime.com, only homemade. The backend is built on django, the queues are made on celery with a backend in redis.

Project domains are added to Goose. For each domain, the addresses to be monitored, the check interval, and the check type are specified. You can check the certificate, content, HTTP headers, redirects, and something else useful.

If something went wrong, Goose can send letters and SMS or call in the middle of the night and explain the situation in a human voice (Twillio service is used for calls and speech synthesis).

The web interface always shows which domains have errors and how the check queue is doing. 20-25 checks are made every minute. Add tags

The functionary, disappointed in the blogger, spoke about the difficult situation at the headquarters.

At the blogger's headquarters Alexei Navalny, which continues to raise funds for the presidential campaign, despite the CEC ban and the explanations of the Constitutional Court, is again a mess. A recent FBK functionary, human rights activist writes about the managerial crisis in the protest ranks on his Facebook page Vitaly Serkuanov. Serukanov explained his departure from Navalny's team by the "need for self-respect."

According to Vitaly Serkuanov, Navalny's headquarters is unable to meet the demands of donors and publish statistics on signatures collected in the regions. The reason for the activist is obvious: the blogger lacks 250,000 votes to be nominated, but getting the necessary number of supporters before the deadline will be an impossible task. The deadline for submitting documents for registration to the CEC ends on January 31, 2018 at 18:00. That is why, as Serukanov notes, Navalny’s team is changing tactics according to the principle Niccolo Machiavelli"End justifies the means". The unsanctioned rallies on December 24 are being planned as a tough transition from campaign failure to future election boycott.

“Volkov (the head of Navalny’s headquarters) did not come up with anything new, except through a sea of ​​detentions, arrests and negative resonance, to avoid answering questions about the reasons for the failure of the campaign, primarily tactical ones. Arouse the sympathy of the masses, fight back with administrative arrests, while ordinary participants will receive criminal terms,” Serukanov writes.

Lawyer Ilya Craft, who conducted an independent investigation into the activities of the FBK, in a comment IA "Politics Today" noted that the number of disillusioned supporters of Navalny is increasing naturally.

The interlocutor of the agency refers former volunteers to the same category. Alexander Turovsky And Denis Lebedev. The first was injured during a search at the blogger's headquarters, but Navalny did not bother to mention his name. A similar story happened to Lebedev three years earlier. His leg was broken on one of the trips on FBK business, but the fund made it clear to the activist that they were not going to deal with his problems.

Craft is sure that Navalny's headquarters are well aware that they will not be able to account for the donations spent and collect the required number of signatures.

“There is really no support. This situation shows that they cannot establish elementary work, because these people have never worked anywhere and have not earned money by honest work. Neither Volkov nor Navalny. So they are drawn to each other. If they had the necessary level of support, then people would pour without stopping. They recruited these signatures even if Volkov and Navalny were completely mediocre, but since they are mediocre and there is no level of support, we get what we get, ”commented Craft.

It is also a story about how, using free software and inexpensive components, a small team created a complex system for collecting signatures across the country. There are no complex technical solutions in the project, but there are many important little things that cannot be foreseen based on typical IT development experience.

For convenience, the material is divided into four posts, which are best read in sequence.

This is technical material, but many of the issues that are discussed here are incomprehensible without a minimum knowledge of the current political context, so it is described to the extent necessary. If for some reason the word “Navalny” scares you (it will appear several more times) or the mention of democratic institutions, just do not read this text. Political issues will not be discussed in the comments.

Campaign goal

Registration of Alexei Navalny as a presidential candidate.

Tasks assigned to the IT department

(in chronological order):

Pre-registration of all who are ready to sign for the nomination of our candidate;
- Ensuring the operation of the network of headquarters throughout Russia;
- Creation of a system for collecting 315 thousand ideal signatures.

Historical and political context

If you do not have a parliamentary party, then you need to collect signatures to participate in elections. This is a defensive procedure that is used to keep "inconsistent" candidates out of the elections.

Endless possibilities for refusal of registration are laid down at the level of collection rules:

• The collection of signatures is strictly limited in time;
• According to the law, a small percentage of the required number of signatures is allocated for marriage, it is impossible to hand over signatures with a good margin;
• It is impossible to verify signatures on your side, since the data of voters must correspond to the FMS database, access to which is available only to state bodies;
• The graphologist, when checking at the CEC, can reject any signature and does not bear legal responsibility in case of an error;
• The verification scheme itself assumes that there will be a significant percentage of false positives (the paradox of Bayes' theorem as a barrier in elections).

We have already encountered this in Novosibirsk when we were collecting signatures for participation in the elections to the Legislative Assembly.

To collect signatures in Novosibirsk, we created the Reaper system, which was focused on collecting signatures "in the field" and on cubes, managed the collectors' routes, took into account all signature lists, and made it possible to rank signatures based on the results of various checks.

Collectors in Novosibirsk brought more than 16,000 signatures, of which we selected and handed in the best 11,722. Despite a tough selection process, the election commission’s working group found many “invalid signatures,” and the election commission refused to register candidates. Read more about the absurd reasons why signatures are invalidated.

The new system was built taking into account the accumulated experience of collecting signatures and their subsequent protection in the election commission.

Features of the new collection of signatures

For the collection of signatures for the nomination of a presidential candidate, even more stringent conditions are established:

No more than 315,000 signatures must be submitted;
- At least 300 thousand signatures must be recognized as valid;
- No more than 7500 signatures are counted from one region;
- For a short collection period (from December 27 to January 31) there are long New Year holidays, when many people go on vacation.

Taking into account previous experience and new requirements, we have adopted the following basic principles.

All-Russian network of headquarters

Due to regional quotas, it was impossible to work in, say, the ten largest cities. 315 thousand signatures could be collected if at least 40 cities were covered. It is more difficult to collect signatures in sparsely populated regions, therefore, in practice, for successful collection, it was necessary to open headquarters in most regions of the country.

The forecast for the number of signatures at the time of the successful completion of the collection shows that in large cities the number of people willing to sign would significantly exceed the regional quotas. Moscow (127 thousand) and St. Petersburg (63 thousand) did not fit on the screen.

Collection of signatures only at headquarters

For house-to-house collection, we would have to hire several thousand pickers. Everyone who has ever worked with paid collectors (or, for example, sociology students) knows that not all of them are equally reverent about the procedure and not everyone overcomes the temptation to simply “draw” a signature or two. Careless filling leads to a large percentage of marriage, and "drawing" signatures is such a common problem that the CEC provides for checking by a graphologist. Even the presence of a graphologist in the state and the exemplary execution of several statements to the police cannot 100% rid the headquarters of the "draughtsmen" (we checked). In addition, the collector can add signatures not only out of malicious intent, but also, vice versa, in order to “help the headquarters”.

We knew that when collecting "in the field" we would definitely introduce "toxic pickers", as was the case in Novosibirsk. Toxic collectors intentionally make mistakes in the voter's data (for example, they replace one digit in the passport number). Their task is to increase the number of invalid signatures above the limit after which the electoral commission refuses to register. Novosibirsk spent a lot of effort to clean up toxic signatures. When collecting throughout the country, this is not possible.

Only in stationary headquarters could it be possible to ensure the sufficient quality of signatures, the conditions for accurately filling out signature sheets and their safety.

Multi-stage signature verification

Perfect signatures are a mathematical abstraction. The real collection of signatures is a complex and difficult process. Even honest and well-trained collectors make mistakes, and given the lack of time, administrative pressure and provocations, there will be even more marriages.

We have a lot of data on how errors appear. According to our experience, in the signature lists collected in a completely honest manner, there will be about 10% of the signatures that the election commission recognizes as invalid.

We had to hand over not just good signatures, but signatures that the election commission would accept. For this, several stages of verification and a ranking mechanism were necessary - in order to select and hand over only those signatures that would most likely pass the checks of the election commission, no matter how absurd we considered them.

Passport scan for each signature

Without a scan, all responsibility for the quality of the signature lies with the assembler. If he accidentally or intentionally made a mistake in the passport number, we will never know.

From experience, we have found that only errors in copying passport data into a signature sheet and data entry errors easily exceed the allowable 5% limit, even if signatures are collected in comfortable conditions and by conscientious collectors.

Having a scan of the document, we could carry out several independent stages of signature verification and make corrections.

In addition, our lawyers were preparing to fight for every signature in court. Last time there was a large category of rejected signatures, about which we knew for sure: the signature matches the passport, but we checked it against an outdated and error-ridden database. A single database and the availability of scans would allow lawyers to automate the process of preparing complaints in such cases.

Of course, it was only possible to scan a passport at headquarters, otherwise it would be impossible to ensure a sufficient level of personal data security.

Synchronization with electronic database

All operations with signatures and subscription lists, all statuses and movements were to be reflected in the electronic database. The signature collection system was supposed to control all stages of collection and detect errors. Only in this way would we maintain order (and mental balance) when working with hundreds of thousands of physical objects.

What has been done in the new version of the system

• In order for us to have somewhere to collect signatures, we have deployed a network of regional headquarters. The IT infrastructure of the headquarters consists of several physical servers, a number of virtual machines, 70 routers, 230 cameras and 189 complete workstations. More than 250 people use the system from the inside at the same time.
• In order to have time to bring several hundred thousand people to headquarters in a short period of time, we started registering voters on the 20!8 website in advance, where they previously confirmed their data.
• To reduce the number of errors, we have created a system that allows independent verification of the correctness of filling out the signature sheet. The system consists of several web applications and a mobile application for two platforms.
• To upload data to the system, we assembled (and partially manufactured) a set of equipment for scanning passports, thought out a scheme for the secure transfer of personal data and implemented it at all headquarters.
• To ensure that the formatting of the address was correct from the point of view of the election commission, we launched a search in the FIAS database and, together with lawyers, seriously tinkered with it to take into account all the requirements of the law.
• In order to (partially) secure the headquarters and have additional arguments in the courts, we have established a 24-hour video surveillance and recording system.
• In order to test the infrastructure, mechanics, clarify the data and prepare the headquarters for the collection, we conducted a large preliminary verification procedure for voters, through which 81,750 people passed.
• We have developed appearance subscription list, a list logistics system at headquarters, and a physical storage and quick access system for the central headquarters.

Core technologies of our web applications

Main backend language: Python.
Frontend: JavaScript, jQuery, React, D3.js.
Frameworks: Django (6 pcs), aiohttp (1 pc).
Database: PostgreSQL, Redis and others.
Full text search: Sphinx.
HTTP server: Nginx, Varnish.
Testing: Jenkins, Browserstack, RobotFramework, Locust.
Monitoring: Zabbix, Elasticsearch, Kibana, Sentry.
Deploy: Ansible and other tools.
Server configuration management: Chef.

Part one: Navalny 20!8 website

We had to bring several hundred thousand people to the headquarters in a very limited period of time. To do this, we started the registration of supporters right on the day the campaign started. Recruiting and registering supporters is one of the main tasks of the Navalny 20!8 website, so there is a registration form on almost every page.

Since all this is needed not just for the sake of beautiful numbers, it was important for us to know that the registered supporters are real people, not bots, to be able to keep in touch with them and understand in which city they are registered (in order to predict the fulfillment of quotas by region). Therefore, registration on the site was quite complicated and required confirmation of the phone number. In order not to deceive ourselves and others, we recorded only people who filled out the entire questionnaire and confirmed their phone number as potential signatories. Therefore, on the main page, instead of a million plus (total number of registrations), we now only have 706,513 “future signatures”.

From the point of view of site building, this is a fairly ordinary product. The site is made on Python + Django + PostgreSQL, standard ORM and standard admin panel are used. For a year and a half, the site has experienced several updates: sections were added, the work of the registration form changed, the texts and images on the pages changed. We tried not to complicate the design so that it was possible to lay out building blocks, thanks to which some sections went from idea to launch in three days.

Approximately half of the visitors to any modern website come from mobile devices. We tried to make the site convenient for everyone, so the layouts were drawn and typeset for correct display on any screen width, starting from 320px.

Headquarters map

The only complex interactive element that visitors see is a map of Russia with headquarters marked on it. When the number of headquarters exceeded 50, it became difficult to navigate the map due to the proximity of markers in the European part of the country. Initially, the map was conceived as a purely decorative element, but suddenly filled with functionality, so for those who have already appreciated the federal nature of the campaign and just want to find their city, we made a list mode.

The map was made using the beautiful and versatile d3.js library. We decided to write our own script, and not use standard Google Maps or Yandex.Maps because of the map projection. There are many ways to unfold the Earth's ellipsoid on a plane. In the Mercator projection, objects are strongly stretched at northern latitudes, and we need more space in those areas where the main big cities. In addition, in the Mercator projection, Russia looks rather strange. We chose the Albers-Siberia conic projection, which is more familiar from geography textbooks.

Russia of a healthy person (Albers conic projection) and Russia of a smoker (Mercator projection)

Content management

The editorial section of the site is of little interest. The usual Django admin panel with minimal customization is used. With limited development resources, it is more profitable to teach several users of the admin panel how to use a standard tool than to spend time creating a really convenient one.

Some solutions that make life easier for the editor were taken from other projects. For example, a tool for typography of texts on the client side. Our typographer is convenient in that it easily connects to any text or string input field. Information about the state of autotypography (on / off) is stored as a non-printable character at the end of the line and does not depend on the backend in any way.

To work with the complex content of posts and news, we use a block editor, which is also used on many other projects:

Blocks are of different types, each project has its own set. Each block contains content and may contain settings. Block data is stored in the database as json, and the markup inside the text block is stored in markdown format.

For display, blocks are converted to the desired format: HTML for a post, text for indexing, RSS or XML for Yandex.Zen, JSON for a mobile application, and so on. Thus, we get a predictable result on any device with fairly complex content formatting.

The first version was based on the Sir Trevor code. Later, when it became difficult to maintain Sir Trevor's spaghetti code, the editor was rewritten in React.

Analytics

The most interesting from a technical point of view takes place in the admin panel of the site. From there we watched the flow of registrations.

At first, the analytics was rather primitive: graphs of the number of registrations different type from time. But we wanted to see the dynamics by region and track the impact of various events on the number of registrations. So the long-awaited analytics appeared:


On this screen there is a summary of information for the entire life of the site, a schedule for a certain period and a list of events for this period. You can highlight a peak on the chart and try to understand what event caused it. Most often, this is the publication of another video with an investigation on Navalny's YouTube channel. The largest increase in signatures was given by videos about the machinations of regional officials.

The chart is made on d3.js, and the filtering of events by time and headquarters is implemented using the Crossfilter library. This solution allows, on the client side, without interface brakes, to operate with registration data for an interval of more than a year in increments of 1 hour. At the moment this is 12 megabytes of data (1.3 MB in gzip).

A small text report with key indicators of the registration base and progress for the previous day was automatically sent to all project participants on a daily basis.

City and region

We also have a huge table, where for each region of Russia the main indicators of preparation for collecting signatures are listed:

The numbers in this table did not want to converge at first. The amount by city was significantly less than the number of registrations. It turned out that when filling out a questionnaire on the site, people unexpectedly often make mistakes in the name of their city or use non-standard names:

Moscow - 2.5% errors and 579 spellings;
- St. Petersburg - 12.6% errors and 767 spellings;
- Komsomolsk-on-Amur - more than 20% of errors and abbreviations, 75 options.

An incorrect estimate of the number of supporters could lead to incorrect planning of the network of headquarters and campaign events. I had to think about how to turn the user input of the city name into the standard name of the region. I didn't want to use auto-completion mechanisms for KLADR or FIAS for such a simple form. Therefore, we took a list of the 700 largest cities in Russia, added a list of typical spellings (“spb”, “n-sk”) and did a non-strict search on them ranking by the Levenshtein distance (this is a measure of the difference between two sets of characters).

We assigned each city from the list to one of three categories according to the distance to the nearest headquarters: the headquarters is in the city, the headquarters is close (urban agglomeration), the headquarters is far. The distance from the headquarters was taken into account when estimating the number of people who would arrive at the right time and sign. In the analytics, we separately counted all signatories and “available” (confirmed mail, lives in the city with headquarters or nearby).

This graph shows how the campaign has become more and more regional over time. The share of new registrations from Moscow and St. Petersburg decreased from 35% to 15%.

SMS and mail

Another technical difficulty was sending SMS and letters. Gateways are not very good at delivering messages, especially to foreign numbers. But we wanted to get the cleanest and most reliable base of supporters, so the second part of the registration form required to verify the phone number via SMS. For reliable sending, we made a rotation of three gateways: if the message was not delivered, then the re-sending went through another gateway. In addition, individual gateways could be turned off when failures occurred on their side. SMS code deliverability is one of the parameters monitored:

The graph shows that there were two failures in the operation of the gateways. The share of delivered SMS dropped sharply on February 21 and April 17-18 due to failures in the message sending queue. And on July 15, we changed the layout of the registration form, this is also noticeable on the chart.

We send a large number of emails to a database of over 700,000 email addresses. Someone subscribed to the news, someone should receive an event notification. In addition, each address must be confirmed according to the 2-opt-in rules (this is when a link is received in the first letter that you need to click on, confirming the subscription to the newsletter). At the beginning of the campaign, we used the ActiveCampaign service, but it is expensive and incredibly slow. When the database exceeded 300 thousand contacts, it became impossible to work. Therefore, we have written our own CRM / mailing service, which allows you to generate mailing lists and chains of letters according to the necessary samples. Mailgun is now used to deliver letters.

Deferred Task Queues

Sending mail or SMS through the API of third-party services is an operation that takes a significant amount of time. Such operations need to be performed asynchronously so as not to slow down the user interface and not put the entire application under load. Initially, all asynchronous tasks worked through Celery with Redis as a broker. Each letter or SMS message created a task in the Celery queue, after which a free worker processed this task. But this approach turned out to be unreliable and too resource-intensive.

Once we received more than 10 thousand registrations in an hour (no, we were not shown on TV, it was a “+1” campaign). 10 Celery workers couldn't handle it, users started to notice a significant delay in receiving SMS and mail.

After this incident, we abandoned Celery in favor of the simplest PostgreSQL-based queue. Tasks from the queue were sorted by “daemons” in python, one for each message delivery channel. Once every 10 seconds, the daemon took a bunch of tasks from the queue and sent data to the mailing API in one packet. Task grouping drastically reduced the load on the server, and the use of a custom queue made debugging and monitoring extremely easy.

Celery turned out to be too complex a tool for our task. It requires thoughtful configuration and monitoring through external utilities like Flower, which itself consumes a lot of resources. On other projects, we try to use a simpler solution - RQ + Redis.

Comparison of the complexity of RQ and Celery from the article about working with asynchronous tasks.

Development process

How is the process of creating the site "Navalny 20! 8" from the point of view of developers? We do not adhere to any one methodology, but use approaches from different systems. For example, managers set tasks in Trello with a structure similar to a kanban board, and developers use certain Extreme Programming practices.

Approximately half of the team is located in the Moscow office, while the rest work remotely. Moscow employees can participate in campaign meetings in order not to work better to understand the big picture, but we discuss the tasks of the IT department separately. Regular calls allow everyone to synchronize and understand the main direction of work at any given time.

Most of the project participants work on it full time, but some tasks were done by developers temporarily involved in other projects, or even volunteers. For example, volunteer Ilya almost completely made a map of headquarters for the main page.

The source code is stored in a git repository on the Bitbucket platform. A separate branch is created for each serious new task. We do not raise a staging server for each branch, they all merge into develop to run on a single test server. After testing, the developer responsible for the task makes a pull request to the master. The team leader looks at the code and, if everything is fine, starts the deployment. For big tasks, developers do detailed descriptions what needs to be checked and what can go wrong during deployment.

Deploy is organized very simply. We have a tool that responds to a webhook from Bitbucket (or a button from its interface), pulls the code from the desired branch, copies it to the server, and runs the update script there. The script is in Makefile.

Running "make update" will update dependencies, migrate, post-process static files, and, if all went well, restart the uwsgi server. We try to make migrations so that they do not break the old code, so in case of deployment errors, everything continues to work.

Development began on September 18, 2016. Since then, there have been 1228 commits, 200 pull requests, deployments over 600 times in production, and 67 branches in the repository (most of them are now closed).

About design

In the project team, only two people constantly worked on the design (an art director with a product function and a designer), while both are actively involved in other projects of the campaign. Therefore, the approach to design was extremely utilitarian.

In the design of IT products, we are always guided by two main principles:

1) Information for the most “lazy” and uninvolved user should be in the most visible place (this is how we, for example, determined the initial places of blocks and sections on the site);

2) The less people will use the final product, the less we try to decorate it (save development resources) and the more input effort we can allow for each user (it is often more efficient to train several people than to spend time implementing new features that will save user effort or avoid mistakes).

Therefore, our low-user internal systems tend to look like a wireframe* that has come to life, and everything that a campaign supporter encounters is part of the overall visual communication, strictly obeys the corporate style and common sense.

An IT system for collecting signatures is a very complex, multi-component project with limited resources, so the main part of the work of designers was on paper, in meetings and in Google docs, and not in a graphic editor (in our case, Sketch).

The project has a lot complex schemes, which you just want to draw, and all the electronic tools for drawing diagrams found on the move did not suit us. Sometimes we used draw.io, but more often we drew directly on paper. The most important diagrams were hung on the project board. Paper “tickets” with questions for discussion at meetings were also attached there.

From paper diagrams and scripts agreed with lawyers, we collected prototypes in marvelapp.com to once again check the logic and make sure nothing was forgotten. Only after that, the layouts were transferred to development.

Depending on the task, different research and design methods were used. So, before doing the long-awaited analytics, we conducted a series of interviews with all potential users of the system (from the chief of staff to the person sending mailing lists) and, based on their wishes, we were able to assemble a very simple interface that served as a campaign dashboard for a long time.

On one page, we saw the flow of registrations, we could see the events that affect it, and find out how our supporters are distributed by city. We also collected ratings of cities by the number of signatories (this allowed us to monitor the effectiveness of headquarters and told us whether we had chosen the right cities for opening new headquarters) and tabular analytics.

For the verification interfaces and the collection of signatures itself, the absolute priority was the speed of the operator. The collection takes place under conditions of acute time pressure, so we tried to save every second and at the same time reduce the number of potential user errors.

According to our calculations, with the existing number of headquarters and under the condition of a continuous flow of people, each picker should have taken no more than 6 minutes per person - from “hello” to the completion of the collection procedure.

Verification and collection of signatures through the IT system is a procedure completely invented by us, so we chose MVP testing on real users of the system as the main method for verifying our solutions. So we tested the basic protocol and the first verification interface on employees of the Moscow headquarters, and then went to three different cities (St. Petersburg, Chelyabinsk and Ulyanovsk) to observe real users in the process. For projects like this The best way quickly make a list of things and user cases that could be forgotten or not foreseen at the design and development stage.

After making minor changes to the interface, verification was launched at all campaign headquarters. As a result, we managed to reduce the processing time of one questionnaire to one and a half to two minutes per person.

Testing

RobotFramework was used for automated testing. To cover the most critical functionality of the project, acceptance and functional tests were written, they were configured automatic start. Jenkins was used as the CI system.

The most important function of the site is user registration, which involves phone confirmation via SMS code. To test messages with codes, a GSM modem with a test SIM card and Asterisk was configured. The SMS code was sent to the mail, from where it was already available for testing.

Found bugs were added to Trello as tasks for developers.

Server infrastructure

The site "Navalny 20!8" continues to work and smoothly becomes the site of the voters' strike campaign, so the information embargo has not yet been lifted, and the story will be short. The server part consists of three levels: backend, caching proxies and edge servers. All configurations are managed through chef, so a server with any role can be quickly raised on a new virtual machine.

The database and application instances work on the backend, each application on its own virtual machine and with its own ip. All servers exist in multiple instances, and the database is replicated in master-slave mode to another machine.

The proxy server has Varnish installed, which deals with caching requests to certain addresses and various url-dependent restrictions. If the backend fails, the site can work indefinitely from a proxy server, only the user registration mechanism will break.

Edge servers are engaged in static caching and ssl termination (then the traffic goes through the VPN network). The essence of these servers is to distribute the bulk of traffic and protect the rest of the infrastructure from attacks. These are weak virtual machines with a gigabit channel in different data centers. The load is distributed by DNS balancing. Edge servers contain a minimum of configuration and, if necessary, can be easily raised in a few minutes. The maximum useful traffic that we had on edge servers was 5 Gb / s for several hours.

Pictures, styles, javascript, json data are stored in such a way that the file name includes a hash of the contents of this file (for example, style.28fa1c7b1761.css), so all these files can be cached forever on the server and in the browser. The main volume of traffic is given from edge servers. Then only requests to content pages pass, and this is about 25 times less data.

Sometimes CloudFlare connects instead of edge servers, but we try to return to our servers, because CloudFlare does not always have good accessibility from Russia. Individual providers, even the largest ones, regularly begin to block their ip (traces of Roskomnadzor).

Conclusion

Collecting signatures in the traditional style (without a special IT system, with paper, pen and spreadsheets in Excel) is like flying a balloon to the moon: yes, if you take enough balloons, you can even fly up and hide in the clouds, but get to targets in this way is physically impossible.

In order to collect such signatures that the election commission will have to accept even from an undesirable candidate, we began to create this complex infrastructure. In this chapter, we talked about the very task set before us, and about preparing for its solution.

It tells about the selection and configuration of headquarters network equipment, the development of your own document scanner and the organization of video surveillance of headquarters premises.

The third chapter will describe the process of creating applications for collecting signatures and everything related to working with physical signature sheets.

The fourth chapter talks about project management, the team, the timeline, and a little about the results.

Tags:

• django
• bulk
• interface design
• website development
• 20!8

Add tags

"While this is a PR story"

Oppositionist Alexei, who collected 300,000 signatures required to register his candidacy in the presidential elections in the Russian Federation in 2018. Will he now be allowed to participate in the elections?

According to the law "On the Election of the President of the Russian Federation" (No. 19 - FZ of 10.01.2003), a candidate is a self-nominated candidate (i.e., the candidate who is not nominated by political party) must provide the Central Election Commission with at least 300,000 signatures of voters, and each region should account for no more than 7,500 of them. Technologically, Navalny completed the task - he announced that he had already collected 335,782 signatures in no less than 40 regions of the country. However, they do not have legal force.

“Signatures collected before the nomination and before the opening of the electoral account are not valid,” Andrey Buzin, an expert in the field of electoral legislation, told us. We explain. According to the same law "On Presidential Elections", the nomination of a candidate takes place no later than 20 days from the date of the official publication of the decision to call elections. While they are scheduled for the anniversary of the annexation of Crimea, March 18, 2018.

However, according to the Constitution, the presidential elections will be officially scheduled by the Federation Council no earlier than 100 days and no later than 90 days before voting day, that is, in December of this year. Accordingly, Alexei Navalny will be able to officially nominate his candidacy for the presidential election and start collecting signatures only in December. If a politician submits documents for nomination on the same day when the Federation Council calls elections, then he will have from 55 to 45 days to collect signatures. If he collects the required number within this period, the CEC will check them for authenticity and make a decision on registration as a candidate.

The fact that the signatures collected now cannot be used when submitting documents to the CEC is also realized by Navalny himself. On his website, he wrote that in order to “collect the right amount quickly and efficiently in X hour”, each signature is an e-mail, a phone number and a short questionnaire of a potential voter.

However, it is too early to talk about Navalny's registration as a candidate. The question of whether he will be able to nominate his candidacy in the presidential elections is still not resolved. After the verdict at the re-examination of the process in the Kirovles case, the oppositionist fell into a legal fork. On the one side? he was sentenced to imprisonment, which means that, according to the law, he has no right to participate in elections. On the other hand, he still has the opportunity to challenge this provision of the law in the Constitutional Court.

The head of the Center for Economic and Political Reforms, Nikolai Mironov, believes that Navalny collected signatures in order to show his political weight, and that he is able to do this in the future.

“This is a way of pressure,” said the MK political scientist. - I think that this is a signal to different social groups so that they support him, and to various sponsors. But we don’t know anything about the quality of these signatures.” The expert noted that a promoted politician can collect 300,000 signatures, but it is very difficult. “As long as this story is outside the electoral process, it is for PR,” the political scientist says. - When real signatures will be collected, everything will depend on election commissions. At some stage, he may be cut off, but, as far as I understand, there is no ready-made scenario for Navalny’s participation in the elections.”