Using Google Vision to Automatically Identify Images across Social Media

We recently attended the Social Customer Service Summit (SCSS16) event in London. It was an excellent event, hosted by Luke Brynley-Jones and attended by many of the great and good within #socialcustcare. Ahead of the event, a challenge went out to social customer service platform providers to see if they could help TFL automatically identify the thousands of images that they receive across Twitter and Facebook every month.

The challenge was to help pre-identify images, relating to different types of transport. Potentially we could recognise certain London landmarks within the images and even look to identify different facial expressions – i.e. happy faces, sad faces etc. Clearly, by automatically recognising the images that customer’s post, we could then automatically allocate them to the correct team, saving manual effort and improving response time.

Well, here at DigiDesk, we like a challenge! So, we looked into various technology and signed up to a BETA programme with Google Vision. The Google Cloud Vision API enables developers to understand the content of an image by encapsulating powerful machine learning models via their API. In short, it allows us to automatically classify images into thousands of categories, matching up with tags and helping to determine sentiment.

We built a test account on DigiDesk, pulling in all of TFL’s public mentions across Twitter and Facebook. Then, by hooking into the Google Vision API, we were able to analyse every public image and…

Automatically Classify Each Image:

Google Vision enabled us to automatically classify each image posted by TFL’s customers with a range of matching ‘tags’. These can be pre-defined as types of transport or London landmarks. We can even bring in tags with the image to help identify certain facial expressions.

Matching up Tag Groups and related Tags:

We set up a range of matching ‘tag groups’ within the DigiDesk account for each category such as, Transport, Landmarks and Facial Expressions. Within each tag group, we then set up specific ‘tags’ to recognise trains, buses, taxis or landmarks and matched up facial expression with happy and sad face tags.

Using Workflow to Automatically Assign Images:

Now that images have been tagged, we can use the tags to automatically assign each image to the correct social team or agent. Workflow within DigiDesk enables images tagged us ‘bus’ to automatically be assigned to a specific team or agent. Whereas, images tagged as ‘taxi’ would go directly to the agent best suited to respond to that customer post.

It’s a pretty powerful piece of workflow. Automatically recognising and classifying images into certain groups and then automatically assigning them to the correct team or agent. This could save many hours of manual classification work and, more importantly, greatly improve response times to customer posts / images. There is also a range of further benefits:

Reporting on the Images we Receive

With each image automatically being tagged us bus, train, taxi etc, we can now easily report on the type and volume of customer posts / images that we receive. This will help us better understand the nature of images we receive and better manage response to those customer posts.

Helping to define the Sentiment of Images

By using Google Vision to help recognise any facial characteristics within the images we receive, we can automatically define the sentiment for each image. Once again, we can use reporting to filter in images for each tag group and monitor / compare sentiment scores for each set of tags.

Using Workflow to Prioritise certain Images

By using a combination of tags that come through Google Vision, we can prioritise certain images for an urgent response. Let’s say we match a certain tag group with an ‘angry face’ – we can then automatically set the priority to ‘high’ and assign to the complaints team to manage quickly and effectively.

Here at DigiDesk, we’ve just started on our journey to automatically classify the images that come in across social channels like Twitter and Facebook. We are keen to continue to explore this area and we are actively looking for more trial / BETA accounts. If you want to take a look at our demo account and discuss further, please click here to contact us and become part of the journey…