I spent some time last night reviewing the logs for the last weeks. I wanted to share with you some numbers about onesoup’s usage so far:

• 513 registered users
• 3,398 chat sessions
• 1,105 upstream login errors (MSN, Yahoo, Gtalk, etc.)
• 401 login errors
• 59,305 messages sent and received

So, while this is good news, it’s also surprising to see so many errors. Almost half the attempts to connect to the service ended up in failure. There is a combination of factors going on here, which I ‘ll take the opportunity to explain.

I consciously decided not to require active registration, but rather to pass on credentials to 3rd party networks. This makes it simpler to use the service, and also avoids me having to store passwords for 3rd party services. I know this is a security feature important for many users.

But passing on credentials has a down-side: not all phones support PWD authentication scheme, and insist on using MD5 (or extremely rarely SHA1). Possibly up to 1/3 of all phones don’t support PWD. That’s a large proportion of users.

I have been thinking how to address this, at the same time that I consider the  move to XMPP. Here’s what I am planning:

• In order to chat through onesoup, users will have to register and pick a username and password. This will allow me to move to support all phones.
• Signing into onesoup will connect you to the onesoup mobile community, on onesoup’s XMPP server.
• As with any XMPP server, users will be able to add XMPP contacts on other servers that are supporting the server-to-server (XMPP federation). Gtalk is allowing it.

By taking these steps, users will be able to chat with other onesoup users, and with any other friends on XMPP. This includes all native XMPP servers with s-2-s enabled, as well as Gtalk and AIM/ICQ.

However, this also means that Yahoo and MSN support will disappear. Yahoo and MSN account today for 284 out of the 513 total users. Even though it’s a difficult decision to make, for me having a reliable service is more important that serving 513 users 50% of the time only. Unfortunately, that means switching off Yahoo and MSN.

Now, these changes won’t happen immediately. I am literally starting to code the new XMPP-powered onesoup IMPS CSP chat service, and it will take weeks until I release anything that works reliably.

I blogged a few weeks ago that IMPS was dead, and that I would be embracing XMPP. Since then I have received many, many, emails from folks asking me to keep the service. I honestly appreciate the support and interest.

I stand by my original posting that IMPS is a failed specification. It is true though that the original arguments for Onesoup, namely being native to the phone and the ability to run on the background, still hold true, and as such, an IMPS service still makes sense.

The issue is that technically IMPS has failed and I am not willing to invest much more time to support a specification that I consider to be broken. I’d much rather spend my time building a mobile IM service on XMPP. But until the manufacturers stop shipping a built-in IMPS client that barely anybody uses, rather than an XMPP client, there is a point in supporting IMPS.

This is the dilemma where I stand:  IMPS is a failed specification yet a widely-spread technology without a short-term viable replacement. My requirement has therefore transformed to providing mobile IM through XMPP, and having the ability to additionally support IMPS at almost no additional cost. The current Onesoup is fully articulated around IMPS, not XMPP, and I need to move away from there.

The current architecture consists of a very light weight Tomcat server implementing the IMPS Data and Client Initiation Request channels. This is fronted by a custom load-balancer written in PHP that examines the XML payload and distributes the request to a particular node based on the session token identifier. Right now there is only one node, but the intent was to be able to scale to multiple nodes this way. The sticky session is unfortunately required to support the CIR STCP (standalone TCP binding), and since the Tomcat server maintains TCP connections to the 3rd party IM services (e.g. AOL, Yahoo, etc.).

In contrast, the proposed architecture is based around the HTTP BOSH binding to XMPP, which avoids having to open bare TCP server sockets. Note that I am selecting ejabberd as a server, which has a built-in http-bind module, but I could use any dedicated BOSH Connection Manager, such as punjab. I’ll decide on this detail as I move further. Also, I am sticking to Apache as web server for now, although for most of the needs it would be more interesting to use nginx as a load balancer, and push the PHP processing to an Apache backend, either through proxy, or a fastcgi processor.

In the proposed architecture, the IMPS data channel support is provided by a stateless IMPS to XMPP BOSH adapter in PHP. The tricky bit is however the CIR channel, which requires the notion of session.

In order to understand how to best support CIR, I decided to scan through the logs looking for CIR usage, and I realized that all devices that support STCP also support SUDP. Unless proven otherwise when I start coding this, I will be dropping support for the STCP binding.

The challenge of moving to XMPP for Onesoup really comes down to implementing a BOSH client in PHP. I’d had loved the device client to help here, but the IMPS specification uses a data channel (HTTP) and a notification channel (TCP, UDP, WAP, SMS) rather than using only long-lived blocking HTTP requests. To adapt IMPS to BOSH, I need then to handle the original IMPS request, and then continue the processing by issuing an asynchronous CURL to the http-bind backend. Since Apache kills long running processes (a good thing), I’ll need some way to catch this and re-launch the request. I’ll also have to avoid holding more than one TCP connection to the http-bind endpoint per client.

I feel this is a step in the right direction. There is certainly further thinking required in terms to solve the CIR. But this architecture would be much easier to adapt going forward into supporting new clients and new applications beyond basic IM.

Over the past few months, I’ve been running this service on alpha for the public. The intent of being alpha was to test the software, and to find out what uptake such service would have. The results are mixed, and I wanted to share them with you.

Our uptake, with absolutely no publicity, has been fantastic. As we speak there are 15 concurrent users chatting using onesoup, and over the last month alone we’ve seen over 200 new users come to onesoup. This is really great. But it’s far from impressive. Yamigo, which was the first free IMPS service provider, built over the years a registered customer base of close to 150,000 users. This may sound a lot, but actually, it’s not.

There are more than 150 million handsets with a built-in IMPS client on them, yet only 0.01% of those are used for chat. That’s a very low penetration rate. There are a few possible explanations for this, operators banning access, expensive data plans, and general lack of consumer appetite for mobile IM — I can list a few reasons why that would be the case, but mainly it’s around the inherent difficulty in maintaining an active communication, and the bad state of the clients –.

The built-in IMPS clients are simply not comparable to what one can get today either from a network provider, e.g. Yahoo! Go, nor aggregators like Fring. Additionally, “features” such as use of XML vs WBXML, lack of support for HTTP CIR bindings, limits in the number of contacts in your roster, and lack of presence integration with the address book, have made the IMPS clients fairly useless and unpopular.

I am convinced that IMPS, Wireless Village, is dead: consumers with IMPS clients do not have a data plan, operators don’t like IMPS, and the native IMPS clients are extremely limiting. This is obvious to me now, but I am sure it has been to the manufacturers for a while. The IMPS v1.3 standard is sort of in a limbo state, and most development is going into other standards, e.g. SIP, XMPP … Manufacturers are not betting on IMPS any longer, and neither is OMA. It’s time to move on, and call it a day.

I strongly believe in mobile communications and that onesoup could provide a great service for users. I have been thinking over the last few months how to evolve onesoup forward. Here’s a few decisions so far:

• I will not write a generic native chat client. I believe this space is now well served with Fring.
• I will not write a generic web chat client. There are many options out there.
• I will not create yet another mobile social network. Nough’ said.
• I will use XMPP.
• I will use geo-location.

So, what could be in the works? The intent is to work on a geo-aware XMPP service. A few ideas so far as of how to execute this:

• Allow connectivity to any XMPP service, and add geo-location extensions on top, i.e. onesoup provides the client, and a BOSH HTTP proxy. This is the closest to the current onesoup service, but on XMPP instead of IMPS.
• Mobile-enable an already established partner, i.e. onesoup provides the client, the BOSH HTTP proxy, and the XMPP servers. The partner provides roster, profiles and authentication.
• Focus on a particular vertical, rather than service general IM needs. Something in my mind is location based mobile dating or games. Again, a partnership with an existing player would be certainly required, or some other means of seeding the system.

What are your thoughts? What would you like to do with mobile messaging?

all services are back to normal now. we only had 60 minutes of downtime, basically for the DNS records to take over. thank you for patience.

heads up guys. we have a new server. the service will be down for the next 2 hours, while we move the service.

today we’ve had a couple of network issues around noon GMT with a faulty router. it’s now fixed and up and running. this is the kind of thing you get with hosting on DSL. but hopefully we’ll move to a new colocation facility within the next few days, where we’ll have redundant kit. for now, please bear with me. thank you guys for your support!

following along the lines of our previous gtalk release, we are now adding support for aim and icq. you can now chat using the embedded chat client in most Nokia, Sony-Ericsson, Siemens or BenQ phones.

behind the scenes in this release, we are also reducing the payload by a factor of 3 for most phones, and a few more phones should be supported now. additionally, not only gtalk, but any jabber account should be supported right now (e.g. myjid@jabber.mobi).

chat client setup instructions here.

again, please give us feedback through the forum.

One of the hard problems to solve with onesoup integration into the native messaging protocols is ensuring messages are actually displayed to the user.

The problem that exists is that onesoup acts as a gateway to the phone, but also as a client to the messaging network. To the network, once a message gets delivered to the client, it has been displayed.

Well, not always. XMPP is the exception and it provides extensions to differentiate between “received” and “processed”. A message might be received many times, but it will only be processed once. IMPS also provides similar acknowledgement capabilities, which are very necessary given the nature of the network. But since this is an “extension”, a draft, to the protocol, we can’t rely on it. Gtalk for example does not use it.

But the problem comes with Yahoo!, AIM/ICQ and MSN (and XMPP until such extension is supported). None of these protocols support transactions, so once onesoup consumes the message, even if the user has not seen it yet, the message won’t be redelivered. Why is this a problem? Let’s use AIM for illustration purposes but this really is the same problem across all of them.

Suppose an AIM message gets delivered to onesoup. AIM will then remove the message from their IM server’s archival offline storage (if the user has one). If the phone fails to pick up the message for whatever reason, and then the user connects via desktop client to the proprietary protocol, the message will be waiting at onesoup. The only time the user will see it is when he or she sign back into onesoup.

To avoid this behavior I am planning to implement at least a couple of features. One is necessary to support the IMPS specs anyway, but the other ones will be an alerting system:

• The spec provides a so-called TimeToLive or KeepAlive, basically the time that needs to pass without any client request for the server to be able to evict the session. It is during this time, until the session times out, where there is a problem of receiving messages and not being able to deliver them if the client has, for example, gone off the cell network. If the KeepAlive setting is too low, besides unnecessary data transfers, one risks frequent disconnections and reconnections, but the chance of messages being kept in the onesoup message store is lower.
• The spec also defines a TCP channel for the server to notify the client that there are messages to collect (CIR). One possibility to limit the risk of storage of undelivered messages is evicting the session if such socket is closed on the client side, and obviously, close the connection to AIM, etc. On cases where the cell phone is re-allocated an IP by the operator, this trick basically means that the phone will need to re-login. But I think this is acceptable, since such behavior is happening daily in our desktop clients due to firewall rules.

But even these two tricks are not enough. If the socket is not closed, there is no guarantee that the client will come to pick up a message after it being notified via CIR. Additionally, not all phones support CIR, so such mechanism won’t work too well for them. But for most phones it should work, assuming the clients behave well-enough to poll right after receiving the CIR.

Ideas or comments appreciated. Should we lower the KeepAlive at the cost of higher data transfers but lower risk of messages sitting on onesoup undelivered?

I still find it surprising that the “classical” messaging network operators don’t support the native (“Wireless Village”) mobile clients shipped with almost every phone nowadays, and they rather prefer to write their own proprietary clients. Besides the development and distribution costs associated, being locked to a proprietary network limits the user’s freedom. In contrast, the native clients are based on a standard specification optimized for mobile networks and mobile phones:

• Standards based means freedom for the user. You can chose between service providers, like Yamigo.
• Optimized for mobile networks means supporting maintaining a session across multiple connections, perhaps even with different IPs. Imagine entering a tunnel, loosing your connection and being able to reestablish the connection and your conversations seamlessly.
• Optimized for mobile phones means things like not draining your batteries by using operator infrastructure like WAP Push or even UDP datagrams.

When thinking Wireless Village, I am really looking at Yamigo, the largest free mobile chat service. Yamigo started in Finland in 2003, developed an OMA IMPS CSP server, formerly known as Wireless Village, and provided a gateway to the mainstream instant messaging networks: Yahoo! messenger, AIM, MSN, and ICQ. Yamigo grew to reach its first 100,000 users in 2006. Since then, there has been little to no development. Even without development and support, Yamigo currently has grown to over 135,000 users.

I strongly think the world needs a service like Yamigo. Personally, I believe we’ll look back at 2008 as the year when mobile IM finally kicked off. Unlike in the past, most operators are now open, and phones ship with better native WV IM clients installed. While some operators still remove the default IM client shipped by the manufacturer, this is becoming less of a norm. In other words, Wireless Village is silently ubiquitous, at least now that Yamigo has been dead for over a year.

It’s to time to wake up! And so I have started the onesoup service, in order to bring a WV service back live. I want the service to be useful and really simple, even if that means only partially supporting the specs. For example, at least initially, onesoup will just be a gateway to the existing networks, and there won’t be any “onesoup network”, meaning no groups, no search, etc.

My goal is to make the experience of using instant messaging on your mobile phone as easy and streamlined as possible. Onesoup will provide automatic registration, and you will be able to access your instant messaging network seamlessly without requiring downloads or complex setups.

I am just starting coding and setting up this blog as I walk through the implementation. I’ll be setting up a mailing list for folks that might be interested in this service, e.g. as private beta testers (since we’ll need to test with a lot of phones!).

Comments or suggestions are highly welcome.