Mobile text messages, logically, really just send data, just like what mobile providers call data, although at a slow rate. My mobile service plan has unlimited text messages but limited data. I was thinking I could use the text message data in place of "data data" for certain things.

I have set up a system, demonstrated on the right, that will let me read the news via text message, no mobile data required. As you will see, it is quite slow, but, this was never intended to be used in a serious manner anyway. The "M" requests the main menu, a number requests a menu item, "N" requests more of the content since only so much content can fit in a single message and I did not want to send an unstoppable flood of messages, and a "$" at the end of the message signifies that there is nothing left to send, that is, you can not send "N" to get more content.

To give a high-level overview, I will start logicalDiagram-small by briefly describing one communication cycle of the system. It is the most interesting part and the rest of the information in this post is just extra rambling for those really interested.

Logical flow of one communication cycle:

  1. The user first sends a command to Twitter via text message. Commands are sent as one Twitter user and received by another. The same user that received the command will later be the one who sends the content back to the mobile phone. The user who sent the command will receive the content. Another way to put it, there is a Twitter account for the phone and another not for the phone.
  2. Twitter will send out an e-mail notification in response to this command.
  3. The e-mail server will send a push alert of the e-mail it received from Twitter and the Python IMAP client will receive this email and go out to the internet and gather content if required.
  4. The Python e-mail client then sends the content to Twitter via the Twitter API and a Twitter direct message.
  5. Twitter receives the direct message and sends out a text message notification.

The system is made up of two main components.

One, a custom Android application which sends and receives SMS messages. Originally this aspect was not thought to be required. I planned to send SMS messages through the email to text message and text message to email services that my mobile service provides. But I found that my service provider seems to rate limit these messages since messages would stop being delivered after sending a handful of them. It seems I could send unlimited messages from text message to e-mail but the rate limit kicked in on the way back (e-mail to text message). I could maybe send 20 before they stopped being delivered. I phoned and messaged my service provider about the situation, they did not know what the problem was and in some cases they did not even know the functionality existed.

Since using the text message to e-mail and e-mail to text message services was ruled out I decided to use use Twitter since it offers text message notifications for things like direct messages. I considered a few other services. I looked at Google, they seem to offer some sort of SMS service but it turns out it is not available for Canada. On a side note, I wonder why this is, I consider Canada to be more technologically advanced than some of the listed countries. I considered Facebook, but Facebook did not seem to reliably deliver text messages. I looked at a few free services but they seemed sketchy. I also looked at Skype.

Twitter is quite reliable but it comes with a few downsides. It adds a bunch of text of its own to every message, letting you know that it is a message from Twitter, who it is from and how to reply to it. As a result, a tweet taking up the entire 140 characters spans across two text messages. This creates two problems. One, it often results in messages arriving out of order, and two, the second message seems to take a longer period of time to arrive, as if again, my service provider is rate limiting me when two text messages are sent to me in a fast fashion. I considered only sending smaller messages so that Twitter could always deliver the message in one text message but I figured this would be slower since relatively more messages would be required since there is less Twitter boilerplate when the message spans two text messages when compared to sending two Twitter messages. This is because Twitter only tells you who the message is from for one Twitter message (even if it spans two text messages), thus a second text message (from the single Twitter message) would not include this boilerplate. If I sent two Twitter messages I'd get two text messages with two sets of boiler plate. Receiving boiler plate, and even messages out of order was not that big of a problem compared to the problems with sending a reply since my eyes can easily filter the boilerplate out, but replies required typing boilerplate. In order to reply to a Twitter direct message via text message you have to write "DM @twitterHandle theMessage." This means that where I send the message "M" to get the main menu I would have had to type "DM @twitterHandle M" which would not be very usable at all. I looked into perhaps finding an SMS Android application with canned responses, macros or snippets, but my searching was not yielding fruitful results.

Thus, I decided to write my first small Android application. This application would parse incoming messages removing the Twitter boilerplate and make sure that messages were ordered properly. This meant that if the second message arrived first the application would wait until the first message arrived before displaying either. For consistency, if the message is multi-part it also waits for the whole message if the first message arrives first. The application also extracts the actual content of the message and displays the message hiding the rest of the information. The application also makes replying easier by automatically inserting the "DM @twitterHandle" aspect.

Two, a custom IMAP client. This is the first component of the system that I wrote. It uses the IDLE functionality to listen to a local self-hosted e-mail server running Dovecot in a virtual machine in order to receive push notifications when e-mail arrives. Once I made the move to Twitter I thought that this could be replaced with the Twitter API but it did not seem like receiving push notifications from Twitter was possible or easy at this point. There is evidence that they are going to make this possible in the future because it seems as if there is a private beta, but for the time being this did not seem like a plausible route. This is okay because the push e-mail notifications work really well anyway. Twitter delivers the email quickly and my IMAP client gets the notifications from the e-mail server nearly instantaneous.

The e-mail client receives notifications from Twitter and parses the e-mail to extract the message, extracting commands like "M" for main menu. It then gathers the content which it needs to send, it makes a few other notes for later, such as when a user enters "N" and requests more of the content and more messages. It then replies to Twitter via a direct message and the Twitter API. The e-mail client has to handle the fact that content which it wishes to send will often be longer than the Twitter maximum of 150 characters.

The gathering content aspect is rather quite neat. In the video I demonstrated it working with CBC Toronto but it is actually very flexible. Getting the CBC Toronto links was hard-coded to a degree but the gathering of the article would be nearly universal to any reasonably coded web page. This means that in the future, though unlikely to be honest (since this was just an experiment), I could add Reddit capability. Reddit would otherwise pose a problem since what it links to is not well defined at all. The way the article content is extracted is by analysing the structure of the web page and guessing which section is the main content. It does this by looking for the div or article tag which contains the most text and the least amount of other tags. This works because the actual content portion of a web page usually has relatively long sections of text without much HTML mark-up. There are a few other conditions to this but that is mostly how it works. Some tags are ignored such as the paragraph tag or the list item tag. This is because the paragraph tag is almost exclusively used for main content and a long list could have many list tags which would raise the tag count significantly. There were also problems like the situation where a div contains one word because that div tag would be 100% comprised of text and thus be very favourable in the text to tag ratio. This happened in situations like

. These problems were dealt with and I was able to successfully test on several web pages chosen at random.

This system because of its speed, and perhaps its complexity makes it not very practical. Aside from the intelligent content extractor, there is one potentially useful part of this experiment, it is making an Android SMS client that facilitates communication with Twitter like I have but takes it further. I could fairly easily group the SMS messages that are received from Twitter by Twitter user and allow the Android user to respond to whoever sent the message (rather than a fixed user as is the case with my application (very simple modification)). This would enable someone to chat on Twitter with their mobile despite not having a data connection. It would behave like a normal SMS application but using Twitter.

It was a fun experiment and proof of concept. I learned a lot, particularly about Android development since I had never done it before. I also figure that Twitter does not like me doing this much so I am unlikely to use this system much if at all. The primary reason I use mobile data is when I am waiting for something, like when at the dentist and waiting for my appointment. This system could be useful if I decided to reduce my data plan because I could occupy the time without using data and instead use text messages. The delay in the system would not be much of a bother since there would be no expectation or real care if it were fast or not. Though perhaps reading a pre-downloaded book would be more useful. Out of curiosity it would be amusing to transmit a small image via text messages.