How I Passed Telus-Internet Assessor, Rater or Appen- Yukon, RWS- Search Engine Evaluator or Welocalize- Scout Search Quality Rater Exam

            I have applied for Internet Assessor role on Lionbridge  in mid of April. Same role is called as Rater for USA applicants, Appen called it as a Project Yukon, RWS called it as a Search Engine Evaluator and Welocalize called it as a Search Quality rater. Exam is same for all these roles as client is Google who conducts exam for this role. Before applying I gathered as much information as possible from Reddit, Quora and Glassdoor. Found that people are working on this position for years and making a decent income. Many people have failed to clear the exam. Many have stated that the exam is the real hurdle for the job. It is true that amongst all the long term roles of search engine evaluator regime, exam of Internet Assessor Role (Rater or Project Yukon) exam is hard. But not impossible to clear. You really need to give attention to detail and spare 30 to 35 hours span over 7 days to clear the exam.

           Immediately after applying receive the confirmation from Lionbridge that my application is submitted. After 2 days they send me the NDA to sign. After signing the NDA immediately on same day I receive the exam mail that my exam will start on 21^st April i.e. after 5 days. Access to the exam portal will be provided on the same day.  I can download the guidelines on 21^st April and need to complete the exam within 7 days.

Following is the abstract of official mail from them.

You are required to use your desktop/ laptop along with your smartphone device to complete the exam.
We recommend that you aim to complete Part 1 within the first two to three days as
Parts 2 and 3 take longer to complete.
You should not complete the Exam in one sitting but take your time over the 7 days.
There is no additional benefit for completing the Exam early, those who perform best
in exam do so by taking it stage by stage over a 7 day period.

How to Best Approach the Exam
Spend time in the Exam Portal before taking the Exam.
In order to pass the Exam you must spend time in the Exam Portal. There you are taught how to answer the Exam questions by taking tutorials.
Do NOT “Release” any exam items.
Do not press the RELEASE button on the top right of your Exam page screen. This will cause you to fail the exam.
Answer each question within 60 minutes.
Do not open any question until you know you have the time to answer it. If you take a question and leave it unanswered longer than this, it will “Expire” and this will cause you to fail the exam.

You should also ensure each question is answered before you close out of each Exam sitting.
Do not dive straight in to complete the Exam on the day you are given access to it.
You will be given access to the Exam and the Exam Guidelines at the same time.
Those who perform best in the exam spend time understanding the Guidelines before attempting their first Exam question.
Ensure you leave at least two days to complete Part 3 of the Exam some find it the most challenging part so not rushing it and studying the guidelines here will be pivotal to your success.

Lionbridge Exam

From above mail it is clear that the Guidelines Document is the key part for the exam I need to study it by heart in order to clear the exam. But guidelines will be available on first day and I have 7 days to study guideline and clear the exam. I searched on the Internet if I get any help on the exam, there is none available. Everyone who has cleared the exam on Reddit forum advised to study the guideline properly. Then I searched for guidelines and found 168 page PDF document just scrolled through the document as not sure if they are same. Later on 21^st April when I received the actual guidelines found that they are the same document. Actually Google itself has made those guidelines available to everyone. I will provide the link to download these guidelines at end of this chapter. The guidelines for only this role (Internet Assessor) of Lionbridge Exam is available publicly. For all the companies/website mention above you need to sign the NDA and no one can share any exam material publicly for any role.

Part 1

As mentioned in mail abstract of chapter earlier exam is having 3 parts. Part 1 is theoretical and Part 2 and part 3 are practical. Again each part is divided in 7 sections and each section is having 1 hour to complete. Once you start any section you need to finish it in 1 hour and if you want you can take a break by clicking on “Submit and Stop” button after answering all the questions in that section. So in all I have 21 hours of exam ahead. This is the longest exam for any of the role. But if you clear it one will get a long term contact position to work for biggest search engine i.e. Google.

            I read the guideline on day one and on day two reread it again it real helps as I found I have missed many important concepts in first reading. Also prepared notes of important points. Though guideline is 175 page long it does not have any technical information it is a simple reading and it contains lots of examples with links. And remember it is open book exam you can keep guidelines open while giving the exam and can use search facility to find exact term. Lionbridge doesn’t mentioned anywhere what is the passing percentage for the exam but it should have been 85% because that is what the quality mark you need to keep while working also. I advised you to keep the target of 90%. Remember Lionbridge doesn’t provide any feedback for the exam. Immediately after completing the Part 1 you will receive the message weather you are successful or not. After passing Part 1 only one can attempt Part 2 and same for Part 3.

Part 1 is theoretical and having 7 sections i.e. you can give exam in 7 in sitting. And each section is having 1 Hour to complete. Each section is having 5 Question and 4 probable statements and you need to mark them as true or false. So 20 statements to answer in each section and last section is having only 3 questions. In this way there are in all 33 questions and 4 statements to each question so you need to answer total 132 statements. I recommend to complete it in single sitting, if you have read guidelines thoroughly. One can easily complete each section within half an hour. But remember people do fail in Part 1 also though Part 1 is easiest part in the exam. The questions are twisted read the question and statements twice keep the guidelines open. The questions are directly from the guidelines. You can copy the statement and find the relevant term in the guideline. Even if you are sure I recommend crosscheck each statement in guideline before answering.

            On Day 3 morning I have attempted Part 1 cross checked each statement in guideline before answering. Completed the part 1 in 3 hours with a small break here and there. Though I am unsure about 5 % of the statements I am confident of clearing it. Immediately after submitting final section I received the mail that I have passed the Part 1 and can proceed to Part 2. 

Part 2 (Page Quality Part)

 Guideline pdf is divided in 3 parts General guidelines, Page Quality (PQ) Guidelines, and Need Meet Rating. In part 2 you will have 7 practical tasks having 1 hour to solve for each task. You can submit and stop exam after completing each task. Rest of the day 3 I have studied Page Quality (PQ) part of the guideline again. Remember there are lots of examples given there with links. This time I have clicked on each link and try to understand reasoning behind the rating.

The goal of PQ rating is to determine how well a webpage achieves its purpose. In order to assign a rating, you must understand the purpose of the webpage and sometimes the website. By understanding the purpose of the webpage, you’ll better understand what criteria are important to consider when evaluating that particular webpage. Websites and webpages should be created to help users. Websites and webpages that are created with intent to harm users, deceive users, or make money with no attempt to help users, should receive the Lowest PQ rating.

There are highest quality and lowest quality webpages of all different types and purposes: shopping pages, news pages, forum pages, video pages, and pages with error messages, PDFs, images, gossip pages, humour pages, homepages, and all other types of pages. The type of page does not determine the PQ rating. You have to understand the purpose of the page to determine the rating.

Pages often make more sense when viewed as part of a website. Some of the criteria in Page Quality rating are based on the website the page belongs to. In order to understand a website, look for information about the website on the website itself. Websites are usually very eager to tell you all about themselves! You must also look for reputation information about the website. We need to find out what outside, independent sources say about the website. When there is disagreement between what the website says about itself and what reputable independent sources say about the website, we’ll trust the independent sources.

The PQ rating is based on how well the page achieves its purpose using the criteria outlined in the guideline.  Lowest, Low, Medium, High, and Highest quality pages. On Page Quality rating tasks, you will use the Page Quality sliding scale (slider) to assign the overall PQ rating. The slider looks like this:

Rating Slider

You may also use the in-between ratings of Lowest+, Low+, Medium+, and High+. Please interpret the “+” as “+ ½,” meaning that the Lowest+ rating is halfway between Lowest and Low, Low+ is halfway between Low and Medium, etc.

Each webpage should be evaluated in terms of 4 different aspects: MC quality, information about the website, Reputation and EAT level of the pages. You need to give the rating for 4 aspects and final rating slider will be automatically adjusted. Guideline suggest that it’s better to examine the pages in terms of 4 different aspects before having an overall PQ judgment however I think that this approach confuses new Raters. So I recommend that you should examine the whole task page and have an overall judgment before proceeding. After that, you can think about quality aspects.

On day 4 I have started attempting Part 2. As mentioned it is having 7 sections of 1 hour each and need to evaluate the PQ rating of 1 Web Page in each section in 1 hour time. The tasks are not from guidelines. But keep guideline open and look for similar examples from the guideline. Generally you will get 1 page each for highest, lowest, low, medium and high quality rating. Most of the results are having tolerance of either + or – or both. That is if you rate the page low+ and actual result is low with tolerance + your rating will still considered correct. Similarly if correct rating for a result is Medium with + and – tolerance, your rating  Medium+ or Low+ will be considered correct. It is not mentioned anywhere in guideline but that’s what we learn while actually rating on the programme in feedbacks.

But remember don’t be generous while giving rating to web pages if you are in confusion go for lower rating. That is if you are doubtful about high rating give Medium+ or if you are in confusion in Low or Medium give in between rating i.e. Low+. Rating is not a science or the art it is just a common sense. Place yourself in role of the user and ask yourself the question will you be satisfied by the result? But also remember personal opinions doesn’t matter here you need to apply the principals of the guidelines which are based on common sense.

Taking considerable break in each section I have completed the Part 2. For 5 tasks I am confident for 2 I am little doubtful and there comes the mail immediately after submitting the 7^th task that I have passed the part 2 and now I can proceed to Part 3.

Part 3 (Needs Met Part)

            Now this is most difficult part of the exam. Most of people fail to clear this part. Even if you fail don’t get disheartened normally Lionbridge offer second chance for the exam which is usually 1 week later. But you need to start again from Part 1 i.e. you need to complete again all the three parts sequentially which is good because you need to apply the knowledge of whole guidelines to clear the Part 3. Here again as mentioned above you need to clear 7 tasks and duration is 1 hour each. You can take break after submitting each task and is highly recommended. Keep 2 full days for Part 3.

            Part 3 is focussed on Needs Met rating. As mentioned above guidelines are having separate Needs Met section. Rest of the day 4 I have gone through Needs Met section again clicked on all the links for the examples given. I have planned to complete 3 tasks on Day 5 and 4 tasks on Day 6, keeping Day 7 reserve.

            Part 3 is totally different basically it is actual work format, now you will be evaluating the whole task. Understanding the query is the first step in evaluating the task. Remember, a query is what a user types or speaks into a mobile phone. Many queries have more than one meaning. For example, the query [apple] might refer to the computer brand or the fruit. We will call these possible meanings query interpretations. The dominant interpretation of a query is what most users mean when they type the query. Not all queries have a dominant interpretation. The dominant interpretation should be clear to you, especially after doing a little web research. A common interpretation of a query is what many or some users mean when they type a query. A query can have multiple common interpretations. Sometimes you will find less common interpretations. These are interpretations that few users have in mind. We will call these minor interpretations. Remember to think about the query and its current meaning as you are rating. We will assume users are looking for current information about a topic, the most recent product model, the most recent occurrence of a recurring event, etc., unless otherwise specified by the query.

            Secondly you need to understand user intent from the query. Many queries have more than one likely user intent. Please use your judgment when trying to decide if one intent is more likely than another intent.

This is what the Needs Met rating slider looks like:

Rating Slider

Fully Meets (FullyM): A special rating category, which only applies to certain queries and results. All or almost all mobile users would be immediately and fully satisfied by the result and would not need to view other results to satisfy their need.

Highly Meets (HM): Very helpful for many or most mobile users. Some users may wish to see additional results.

Moderately Meets (MM): Helpful for many users OR very helpful for some mobile users. Some or many users may wish to see additional results.

Slightly Meets (SM): Helpful for fewer mobile users. There is a connection between the query and the result, but not a strong or satisfying connection. Many or most users would wish to see additional results.

Fails to Meet (FailsM): Completely fails to meet the needs of the mobile users. All or almost all users would wish to see additional results.

            The Needs Met rating is based on both the query and the result. You must carefully think about the query and user intent when assigning a Needs Met rating. The Page Quality rating slider does not depend on the query. Do not think about the query when assigning a Page Quality rating to the LP (Landing Page, Web Page). Some results don’t have a Page Quality slider. If a result block has no Page Quality rating slider, you do not have to give a Page Quality rating. If there is a Page Quality slider, please assign a Page Quality rating based on the landing page.

            You need to assign Porn, Foreign Language, Did Not Load, and Upsetting-Offensive flags to result blocks when appropriate. Some rating tasks may ask you to identify Not-for-Everyone results. All flags are query-independent, meaning that they do not depend on the query.

            Some queries really only have one meaning. Consider the query [iphone], English (US). There may be different user intents for this query (research iPhones, buy an iPhone, go to the iPhone page on Apple’s website), but all users are basically referring to the same thing: the phone made by Apple, Inc. Some queries truly have different possible meanings. Consider the query [apple], English (US). Some users may want to find more information on the computer brand or the fruit. We refer to these different meanings as query interpretations. When giving Needs Met ratings for results involving different query interpretations, think about how likely the query interpretation is and how helpful the result is.

            A very helpful result for a dominant interpretation should be rated Highly Meets, because it is very helpful for many or most users. Some queries with a dominant interpretation have a FullyM result. A very helpful result for a common interpretation may be Highly Meets or Moderately Meets, depending on how likely the interpretation is. A very helpful result for a very minor interpretation may be Slightly Meets or lower because few users may be interested in that interpretation. There are some interpretations that are so unlikely that results should be rated FailsM.

            It’s probably the hardest part of the exam. First of all, you need to analyse queries to understand user intents and needs fully because ratings should be given according to the user’s intents. I think people fail this part because they are approaching the tasks ignoring the user intents and needs. A result cannot be automatically useful because of its relevance to the query. You should think like you are the person who wrote that query.

In this part of the exam, each task consists of 8-10 result blocks to rate their both Page Quality and Needs Met, within 1 hour. Remember in Part 2 we are giving PQ rating for single web page and there is no query. Here we need to understand query first and need to give PQ rating which is independent of query or its intent and Needs Met rating which depend on understanding query and user intent. So in all you are evaluating 8 to 10 web pages for PQ and Needs Met rating.

Many people fail in the Part 3 because they are not ready for such a huge task to complete as compare to Part 2. As it is not mentioned anywhere in guideline or in the exam mail. So one need to mentally prepare for this before starting on Part 3. But is it really a huge task? Answer is no. When you will actually start working you will be rating such a task in 10 minute. Here you have 1 full hour. Just keep in mind that there are many things to do in Part 3 as compare to Part 2. Secondly don’t wait for full 1 hour to submit the task. As you are rating 8 to 10 blocks it may happens that 1 or 2 ratings are forgotten to mark. Submit the task on 55 or 56 minute on timer as it may take 1 or 2 minute to submit rating if anything is incomplete the task will not submit and it will highlight that rating in red square. If you fail to submit the task within hour it will get auto submitted. Auto submitted tasks are not considered for evaluation and you will lose the whole marks for that task.

In each task you will have all type of PQ rating as mentioned earlier and all type of Needs Met rating. i.e FullyM, MM, MM, SM, and FaislM. You can also give in between rating. Don’t be generous if in doubt give lower rating. If in doubt between two ratings give in between rating. Most of the results are having tolerance of + or – or both. But here rules are stricter as compare to PQ rating. Fully met is always fully met and no tolerance almost in all cases. Similarly Fails to meet is always fails to meet in almost all cases.

Key to pass Part 3 is understanding Needs Met properly and be ready to evaluate 8 to 10 result blocks in a hour time. I have completed Part 3 on Day 5 and Day6 taking 1 to 2 hour gap in each task. After completing Part 3 you will not get result immediately, if your pass. If you fail you will get mail that you are not successful immediately. But usually they give you second chance and you need to go through all the parts again. If you don’t get any mail, need to wait for 2 to 3 days after exam period is over and then you will notified that you have achieved overall pass they did not share feedback or actual marks.

I received the welcome mail on 30^th April. And I started working on 1^st of May. It takes 15 days from the application to actual starting working as a Internet Assessor.

While Working

You are allowed to maximum 20 hours per week and minimum 10 hours per week is mandatory when work is available. Sometimes you are allowed to work for 24 to 30 hours per week. I have received this opportunity more than 10 times in a year. There are times when work availability is low and you can’t get full quota of work.

Initially working can be overwhelming you need to grasp many things. But remember you have learned most difficult part of your work while preparing for exam. They will also consider you for first one or two month. You will get used it within a month. In actual work you get many different types of task. Most of them are super easy and fun to work on. Like you will get YouTube video to rate. Audio files to judge for the quality. Every task is having estimated time tagged with it. Remember here you are paid on hourly basis and not on task basis. If you rush through the tasks you will only reducing your chances of getting full quota of work. Vice versa if you have taken more than estimated time for completing task your productivity will be hampered and you will warned for such a practice. Your time sheet can be sent back for corrections. Thumb rule is report time based on estimated time of each task however around 5% variance is acceptable.

Once or twice in a month there will be blind test. Blind test will be based on the task type which are appeared in exam. So guidelines are important while working as well. Guidelines are your rule book. You need to maintain the quality of 85% for first year of your work barring first 3 months of your work. First year onward it is 87%. You will get feedback on blind test every month and you are expected to improve. Continuous bad feedback can kept you on review and you will get very few task and that too are blind test task still you will improve your result to expected level. Sometimes they give you additional learning path and once cleared you will be normal to work on. In month or two by looking at the task you can be able to know whether it is a blind test or not and can proceed with extra care. But it is advisable to consider every task as a blind test and be faithful to yourself while giving judgement. If you know guidelines by heart your judgement will be always in line. If you have passed such a tough exam then working is not going to be difficult. There is no harm in going back to guideline and refreshing your knowledge once in a while.

How Do I Get Paid

Pay rate is around $5 per hour for Africa And Asia, $15 per hour for USA, $18 per hour for European countries. Lionbridge pay by direct payment transfer. Amount will be credited by invoice + 30 days. That is to say your January month payment you will receive at the end of February. Even if you leave the program you will be receiving all the dues on expected dates.

Exam Assistance – On-job Support

We provide assistance to Telus Assessor/ Rater Exam, Internet Safety Evaluator Welocalize Search Rater Exam, RWS Search Engine Evaluator Exam, Appen Connect  Yukon Exam

If you need paid consultation or assistance on Exam/ on-job. Please write to me at techsurbhi@gmail.com or WhatsApp/ Telegram at +919960622622 for guaranteed results and on-job support

Download Guidelines:

Search “General Guidelines – Raterhub guidelines.raterhub.com” in google as the direct link didn’t get hyperlinked in this book for the same.

Application Link For USA Rater : Telus Careers Page

Application Link For Rest of the World: Telus Careers Page

Application Link for Appen Connect: Sign up & Look for Project Yukon

Application Link For Welocalize Rater : *Scout Search Quality Rater

Application Link For RWS : Search Engine Evaluator