I applied for the Internet Assessor role on Lionbridge (the same role is named Rater for USA applicants, Appen called it a Project Yukon, Welocalize called it a Search Quality rater and RWS called it a Serch Engine Evaluator. The exam is the same for all these roles as client is Google who conducts the exam for this role.) in mid-April. Before applying I gathered as much information as possible from Reddit, Quora, and Glassdoor. Found that people are working in this position for years and making a decent income. Many people have failed to clear the exam. Many have stated that the exam is the real hurdle for the job. It is true that for all the above roles mentioned exam of Internet Assessor Role (Rater) exam is hard. But not impossible to clear. You really need to give attention to detail and spare 30 to 35 hours over 7 days to clear the exam.
Immediately after applying to receive confirmation from Lionbridge that my application is submitted. After 2 days they send me the NDA to sign. After signing the NDA immediately on the same day I receive the exam mail that my exam will start on 21st April i.e. after 5 days. Access to the exam portal will be provided on the same day. I can download the guidelines on 21st April and need to complete the exam within 7 days.
Following is the abstract of the official mail from them.
You are required to use your desktop/ laptop along with your smartphone device to complete the exam.
We recommend that you aim to complete Part 1 within the first two to three days as
Parts 2 and 3 take longer to complete.
You should not complete the Exam in one sitting but take your time over the 7 days.
There is no additional benefit for completing the Exam early, those who perform best
in the exam do so by taking it stage by stage over a 7-day period.
How to Best Approach the Exam
Spend time in the Exam Portal before taking the Exam.
In order to pass the Exam you must spend time in the Exam Portal. There you are taught how to answer the Exam questions by taking tutorials.
Do NOT “Release” any exam items.
Do not press the RELEASE button on the top right of your Exam page screen. This will cause you to fail the exam.
Answer each question within 30 minutes.
Do not open any question until you know you have the time to answer it. If you take a question and leave it unanswered longer than this, it will “Expire” and this will cause you to fail the exam.
You should also ensure each question is answered before you close out of each Exam sitting.
Do not dive straight in to complete the Exam on the day you are given access to it.
You will be given access to the Exam and the Exam Guidelines at the same time.
Those who perform best in the exam spend time understanding the Guidelines before attempting their first Exam question.
Ensure you leave at least two days to complete Part 3 of the Exam some find it the most challenging part so not rushing it and studying the guidelines here will be pivotal to your success.
TELUS International (Formerly Lionbridge) Exam
From the above mail, it is clear that the Guidelines Document is the key part of the exam I need to study it by heart in order to clear the exam. But guidelines will be available on the first day and I have 7 days to study guidelines and clear the exam. I searched on the Internet if I get any help on the exam, but there is none available. Everyone who has cleared the exam on the Reddit forum is advised to study the guideline properly. Then I searched for guidelines and found 168 page PDF document just scrolled through the document as not sure if they are the same. Later on, on 21st April when I received the actual guidelines found that they are the same document. Actually, Google itself has made those guidelines available to everyone. I will provide the link to download these guidelines at end of this chapter. The guidelines for only this role (Internet Assessor) of the Lionbridge Exam are available publicly. For all the companies/websites mentioned above you need to sign the NDA and no one can share any exam material publicly for any role.
Part 1
As mentioned in the mail abstract of the chapter earlier exam is having 3 parts. Part 1 is theoretical and Part 2 and part 3 are practical. Again each part is divided into 7 sections and each section is having 0.5 hours to complete. Once you start any section you need to finish it in 0.5 hours and if you want you can take a break by clicking on the “Submit and Stop” button after answering all the questions in that section. So in all, I have 10 hours of exams ahead. This is the longest exam for any of the roles. But if you clear it one will get a long-term contract position to work for the biggest search engine i.e. Google.
I read the guideline on day one and on day two reread it again it really helps as I found I missed many important concepts in the first reading. Also prepared notes of important points. Though the guideline is 175 pages long it does not have any technical information it is a simple reading and it contains lots of examples with links. And remember it is an open-book exam you can keep guidelines open while giving the exam and can use a search facility to find the exact term. Lionbridge doesn’t mention anywhere what is the passing percentage for the exam but it should have been 85% because that is what the quality mark you need to keep while working also. I advised you to keep the target of 90%. Remember Lionbridge doesn’t provide any feedback for the exam. Immediately after completing Part 1, you will receive a message about whether you are successful or not. After passing Part 1 only one can attempt Part 2 and the same for Part 3.
Part 1 is theoretical and has 10 sections i.e. you can give an exam in 10 sittings. And each section is having 0.5 Hours to complete. Each section is having 3 Questions and 4 probable statements and you need to mark them as true or false. So 12 statements to answer in each section. In this way, there are all 30 questions and 120 statements. I recommend completing it in a single sitting if you have read the guidelines thoroughly. One can easily complete each section within half an hour. But remember people do fail in Part 1 also though Part 1 is the easiest part of the exam. The questions are twisted read the question and statements twice to keep the guidelines open. The questions are directly from the guidelines. You can copy the statement and find the relevant term in the guideline. Even if you are sure I recommend cross-checking each statement in the guidelines before answering.
On Day 3 morning, I attempted Part 1 cross-checking each statement in the guideline before answering. Completed part 1 in 3 hours with a small break here and there. Though I am unsure about 5 % of the statements I am confident of clearing it. Immediately after submitting the final section, I received the mail that I have passed Part 1 and can proceed to Part 2.
Part 2 (Page Quality Part)
Guideline pdf is divided into 3 parts General guidelines, Page Quality (PQ) Guidelines, and Need Meet Rating. In part 2 you will have 7 practical tasks having 1 hour to solve for each task. You can submit and stop the exam after completing each task. For rest of the day 3, I studied Page Quality (PQ) part of the guideline again. Remember there are lots of examples given there with links. This time I clicked on each link and try to understand the reasoning behind the rating.
The goal of the PQ rating is to determine how well a webpage achieves its purpose. In order to assign a rating, you must understand the purpose of the webpage and sometimes the website. By understanding the purpose of the webpage, you’ll better understand what criteria are important to consider when evaluating that particular webpage. Websites and webpages should be created to help users. Websites and webpages that are created with the intent to harm users, deceive users, or make money with no attempt to help users, should receive the Lowest PQ rating.
There are the highest quality and lowest quality webpages of all different types and purposes: shopping pages, news pages, forum pages, video pages, pages with error messages, PDFs, images, gossip pages, humor pages, homepages, and all other types of pages. The type of page does not determine the PQ rating. You have to understand the purpose of the page to determine the rating.
Pages often make more sense when viewed as part of a website. Some of the criteria in Page Quality rating are based on the website the page belongs to. In order to understand a website, look for information about the website on the website itself. Websites are usually very eager to tell you all about themselves! You must also look for reputation information about the website. We need to find out what outside, independent sources say about the website. When there is a disagreement between what the website says about itself and what reputable independent sources say about the website, we’ll trust the independent sources.
The PQ rating is based on how well the page achieves its purpose using the criteria outlined in the guideline. Lowest, Low, Medium, High, and Highest quality pages. On-Page Quality rating tasks, you will use the Page Quality sliding scale (slider) to assign the overall PQ rating. The slider looks like this:
You may also use the in-between ratings of Lowest+, Low+, Medium+, and High+. Please interpret the “+” as “+ ½,” meaning that the Lowest+ rating is halfway between Lowest and Low, Low+ is halfway between Low and Medium, etc.
Each webpage should be evaluated in terms of 4 different aspects: MC quality, information about the website, Reputation and EAT level of the pages. You need to give the rating for 4 aspects and the final rating slider will be automatically adjusted. The guideline suggests that it’s better to examine the pages in terms of 4 different aspects before having an overall PQ judgment however I think that this approach confuses new Raters. So I recommend that you should examine the whole task page and have an overall judgment before proceeding. After that, you can think about quality aspects.
On day 4 I started attempting Part 2. As mentioned it is having 7 sections of 0.5 hours each and needs to evaluate the PQ rating of 1 Web Page in each section in 0.5 hour time. The tasks are not from guidelines. But keep guidelines open and look for similar examples from the guideline. Generally, you will get 1 page each for the highest, lowest, low, medium, and high-quality ratings. Most of the results are having a tolerance of either + or – or both. That is if you rate the page low+ and the actual result is low with tolerance + your rating will still be considered correct. Similarly, if the correct rating for a result is Medium with + and – tolerance, your rating Medium+ or Low+ will be regarded as correct. It is not mentioned anywhere in the guidelines but that’s what we learn while actually rating the program in feedback.
But remember don’t be generous while giving ratings to web pages if you are in confusion go for the lower rating. That is if you are doubtful about a high rating give Medium+ or if you are in confusion about Low or Medium give in-between ratings i.e. Low+. Rating is not a science or art it is just common sense. Place yourself in the role of the user and ask yourself the question will you be satisfied by the result? But also remember personal opinions don’t matter here you need to apply the principles of the guidelines which are based on common sense.
Taking considerable breaks in each section I have completed Part 2. For the 5 tasks I am confident for 2, I am a little doubtful, and there comes the mail immediately after submitting the 7th task that I have passed part 2 and now I can proceed to Part 3.
Part 3 (Needs Met Part)
Now, this is the most difficult part of the exam. Most people fail to clear this part. Even if you fail don’t get disheartened normally Lionbridge offers a second chance for the exam which is usually 1 week later. But you need to start again from Part 1 i.e. you need to complete again all three parts sequentially which is good because you need to apply the knowledge of the whole guidelines to clear Part 3. Here again, as mentioned above you need to clear 7 tasks and the duration is 0.5 hours each. You can take a break after submitting each task and is highly recommended. Keep 2 full days for Part 3.
Part 3 is focussed on the Needs Met rating. As mentioned above guidelines are having separate Needs Met section. For rest of the day 4, I have gone through the Needs Met section again and clicked on all the links for the examples given. I have planned to complete 3 tasks on Day 5 and 4 tasks on Day 6, keeping Day 7 reserved.
Part 3 is totally different basically it is the actual work format, now you will be evaluating the whole task. Understanding the query is the first step in evaluating the task. Remember, a query is what a user types or speaks into a mobile phone. Many queries have more than one meaning. For example, the query [apple] might refer to the computer brand or the fruit. We will call these possible meanings query interpretations. The dominant interpretation of a query is what most users mean when they type the query. Not all queries have a dominant interpretation. The dominant interpretation should be clear to you, especially after doing a little web research. A common interpretation of a query is what many or some users mean when they type a query. A query can have multiple common interpretations. Sometimes you will find less common interpretations. These are interpretations that few users have in mind. We will call these minor interpretations. Remember to think about the query and its current meaning as you are rating. We will assume users are looking for current information about a topic, the most recent product model, the most recent occurrence of a recurring event, etc. unless otherwise specified by the query.
Secondly, you need to understand user intent from the query. Many queries have more than one likely user intent. Please use your judgment when trying to decide if one intent is more likely than another intent.
This is what the Needs Met rating slider looks like:
Fully Meets (FullyM): A special rating category, which only applies to certain queries and results. All or almost all mobile users would be immediately and fully satisfied by the result and would not need to view other results to satisfy their needs.
Highly Meets (HM): Very helpful for many or most mobile users. Some users may wish to see additional results.
Moderately Meets (MM): Helpful for many users OR very helpful for some mobile users. Some or many users may wish to see additional results.
Slightly Meets (SM): Helpful for fewer mobile users. There is a connection between the query and the result, but not a strong or satisfying connection. Many or most users would wish to see additional results.
Fails to Meet (FailsM): Completely fails to meet the needs of mobile users. All or almost all users would wish to see additional results.
The Needs Met rating is based on both the query and the result. You must carefully think about the query and user intent when assigning a Needs Met rating. The Page Quality rating slider does not depend on the query. Do not think about the query when assigning a Page Quality rating to the LP (Landing Page, Web Page). Some results don’t have a Page Quality slider. If a result block has no Page Quality rating slider, you do not have to give a Page Quality rating. If there is a Page Quality slider, please assign a Page Quality rating based on the landing page.
You need to assign Porn, Foreign Language, Did Not Load, and Upsetting-Offensive flags to result from blocks when appropriate. Some rating tasks may ask you to identify Not-for-Everyone results. All flags are query-independent, meaning that they do not depend on the query.
Some queries really only have one meaning. Consider the query [iphone], English (US). There may be different user intents for this query (research iPhones, buy an iPhone, go to the iPhone page on Apple’s website), but all users are basically referring to the same thing: the phone made by Apple, Inc. Some queries truly have different possible meanings. Consider the query [apple], English (US). Some users may want to find more information on the computer brand or the fruit. We refer to these different meanings as query interpretations. When giving Needs Met ratings for results involving different query interpretations, think about how likely the query interpretation is and how helpful the result is.
A very helpful result for a dominant interpretation should be rated Highly Meets because it is very helpful for many or most users. Some queries with a dominant interpretation have a FullyM result. A very helpful result for a common interpretation may be Highly Meets or Moderately Meets, depending on how likely the interpretation is. A very helpful result for a very minor interpretation may be Slightly Meets or lower because few users may be interested in that interpretation. There are some interpretations that are so unlikely that results should be rated FailsM.
It’s probably the hardest part of the exam. First of all, you need to analyze queries to understand user intents and needs fully because ratings should be given according to the user’s intents. I think people fail this part because they are approaching the tasks ignoring the user’s intents and needs. A result cannot be automatically useful because of its relevance to the query. You should think like you are the person who wrote that query.
In this part of the exam, each task consists of 8-10 result blocks to rate their both Page Quality and Needs Met, within 0.5 hours. Remember in Part 2 we are giving PQ rating for a single web page and there is no query. Here we need to understand the query first and need to give a PQ rating which is independent of the query or its intent and a Needs Met rating which depends on the understanding query and user intent. So in all, you are evaluating 8 to 10 web pages for PQ and Needs Met rating.
Many people fail in Part 3 because they are not ready for such a huge task to complete as compared to Part 2. As it is not mentioned anywhere in the guidelines or in the exam mail. So one needs to mentally prepare for this before starting on Part 3. But is it really a huge task? The answer is no. When you will actually start working you will be rating such a task in 10 minutes. Here you have 0.5 hours. Just keep in mind that there are many things to do in Part 3 as compared to Part 2. Secondly don’t wait for a full 0.5 hours to submit the task. As you are rating 8 to 10 blocks it may happen that 1 or 2 ratings are forgotten to mark. Submit the task in 25 or 26 minutes on a timer as it may take 1 or 2 minutes to submit a rating if anything is incomplete the task will not submit and it will highlight that rating in red square. If you fail to submit the task within 0.5 hours it will get auto-submitted. Auto-submitted tasks are not considered for evaluation and you will lose the whole mark for that task.
In each task, you will have all types of PQ ratings as mentioned earlier and all types of Needs Met ratings. i.e FullyM, MM, MM, SM, and FaislM. You can also give in-between ratings. Don’t be generous if in doubt give a lower rating. If in doubt between two ratings give in between ratings. Most of the results are having a tolerance of + or – or both. But here rules are stricter as compared to the PQ rating. Fully met is always fully met and no tolerance almost in all cases. Similarly Fails to meet is always fails to meet in almost all cases.
Key to passing Part 3 is understanding Needs Met properly and being ready to evaluate 8 to 10 result blocks in an hour’s time. I have completed Part 3 on Day 5 and Day 6 taking a 1 to the 2-hour gap in each task. After completing Part 3 you will not get results immediately if your pass. If you fail you will get mail that you are not successful immediately. But usually, they give you second chance and you need to go through all the parts again. If you don’t get any mail, need to wait for 2 to 3 days after the exam period is over, and then you will be notified that you have achieved an overall pass they did not share feedback or actual marks.
I received the welcome mail on 30th April. And I started working on 1st of May. It takes 15 days from the application to actually start working as an Internet Assessor.
While Working
You are allowed a maximum of 20 hours per week and a minimum of 10 hours per week is mandatory when work is available. Sometimes you are allowed to work for 25 to 30 hours per week. I have received this opportunity more than 10 times a year. There are times when work availability is low and you can’t get the full quota of work.
Initially working can be overwhelming you need to grasp many things. But remember you have learned the most difficult part of your work while preparing for the exam. They will also consider you for the first one or two months. You will get used to it within a month. In actual work, you get many different types of tasks. Most of them are super easy and fun to work on. You will get YouTube videos to rate. Audio files to judge for quality. Every task is having estimated time tagged with it. Remember that you are paid on an hourly basis and not on a task basis. If you rush through the tasks you will only reduce your chances of getting a full quota of work. Vice versa if you have taken more than the estimated time for completing the task your productivity will be hampered and you will be warned for such a practice. Your time sheet can be sent back for corrections. The thumb rule is to report time based on the estimated time of each task however around 5% variance is acceptable.
Once or twice a month, there will be a blind test. The blind test will be based on the task type which is appeared in the exam. So guidelines are important while working as well. Guidelines are your rule book. You need to maintain a quality of 85% for the first year of your work barring the first 3 months of your work. First-year onward it is 87%. You will get feedback on the blind test every month and you are expected to improve. Continuous bad feedback can keep you on review and you will get very few tasks and that too are blind test tasks still you will improve your result to the expected level. Sometimes they give you an additional learning path and once cleared you will be normal to work on. In a month or two by looking at the task you can be able to know whether it is a blind test or not and can proceed with extra care. But it is advisable to consider every task as a blind test and be faithful to yourself while giving judgment. If you know guidelines by heart your judgment will be always in line. If you have passed such a tough exam then working is not going to be difficult. There is no harm in going back to guidelines and refreshing your knowledge once in a while.
How Do I Get Paid
Telus pay by direct payment transfer. The amount will be credited by invoice + 30 days. That is to say your January month payment you will receive between the 10th -15th of March. Even if you leave the program you will be receiving all the dues on the expected dates.
Exam Assistance – On-job Support
We provide assistance to Telus Assessor/ Rater Exam, Internet Safety Evaluator Welocalize Search Rater Exam, RWS Search Engine Evaluator Exam, Appen Connect Yukon Exam
If you need paid consultation or assistance on Exam/ on-job. Please write to me at techsurbhi@gmail.com or WhatsApp/ Telegram at +919960622622 for guaranteed results and on-job support
Download Guidelines:
Search “General Guidelines – Raterhub guidelines.raterhub.com” in Google as the direct link didn’t get hyperlinked in this book for the same.
Application Link For All Telus : Telus Careers Page
Application Link for Appen Connect: Sign up & Look for Project Yukon
Application Link For Appen USA Rater : Raterlabs Careers Page
Application Link For Welocalize Rater : *Scout Search Quality Rater
Application Link For RWS : Search Engine Evaluator
