Trendyol
Today, Trendyol a well known Turkish e-commerce platform and Google Cloud partner will discuss how it leverages Looker to enhance its BigQuery and BigLake data lake infrastructure, enabling thousands of users to obtain real-time insights while upholding strict cloud governance.
The best thing about cloud environments is that, as opposed to a “one size fits all” approach, you may select the services you require and get started right away. However, maintaining enterprise-wide best practices and compliance like ISO27001 can occasionally be difficult, leading to extra authorization needs and conflict within teams. Due to this inherent contradiction, those using cloud resources and those tasked with ensuring the organization’s overall well-being may have conflicting priorities. As a matter of fact, code review procedures and strict security guidelines are said to be able to considerably slow down developers by 69% of organisations.
In light of this, let’s examine how top e-commerce platform Trendyol achieves cloud governance at scale by visualising petabytes of data absorbed from hundreds of sources using Looker.
Innovation and risk reduction in balance
Cloud computing is rarely used by a single development team within an organisation; instead, it is a complicated jumble of internal, production, and non-production apps, each competing for attention and with its own special requirements. It’s also critical to keep in mind that there may be involvement from a variety of governance bodies, such as audit teams, IT, and organisational governance. As such, you must find solutions that satisfy everyone.
Let’s investigate the two opposing viewpoints:
Freedom
The ability to explore and create is something most developers long for. New concepts must be rapidly prototyped and put into production. They are reluctant to make adjustments if development cycles take longer.
Governance
To guarantee that systems are safe, dependable, and expandable, governance is required. This entails doing security assessments, enforcing coding standards, and keeping an eye on system performance particularly with regard to resources that are shared across many teams.
In cloud contexts, these dynamics may provide more difficulties. For example, thousands of people at Trendyol utilise Looker for routine business intelligence tasks, such as processing data and generating reports that aid in streamlining operations, customising user experiences, and forecasting market trends. Although data analysts make up the majority, Looker is used by many other sorts of users inside the organisation, such as business users, warehouse operations, and freight operations.
All of these customers want to use data in their own unique ways to accelerate their goals and change the way they operate, which makes sense. For example, they want to set up periodic schedules to transfer data from Looker to Google Sheets. Although these tools offer effective means of obtaining insights, Trendyol must also make sure that these actions are carried out justly and effectively to prevent negatively affecting experience or resulting in needless expenses.
With the adoption of cloud services, many organisations find themselves in a similar situation. Teams can outsource the labor-intensive tasks and concentrate on more important business priorities thanks to cloud capabilities. Despite their best efforts, they might not always be cautious or fully aware of the risks, which could have a negative effect on the current business.
Therefore, in order to develop systems that support productivity while still being safe and dependable, it’s imperative to strike the correct balance between innovation and risk reduction.
Using Looker to create e-commerce experiences that set trends
Trendyol wants to make its platform quick, safe, and inexpensive while enabling users to explore and find fresh perspectives. This is a task that calls for teamwork and strategic thought in addition to technology.
Trendyol can centralise important metrics from its current platforms and technologies with Looker in almost real time, giving its business users, data analysts, and other teams access to more efficient data analytics and insights to support everyday processes. Additionally, Looker offers simple methods for implementing data governance, such as allowlists, job scheduling, cache optimisations, and access control features. Using Looker, Trendyol’s data warehousing team developed a set of internal policies and procedures to guarantee cloud governance and preserve flexibility. Among them were the following:
- Establishing a formal allowlist. Trendyol proactively created and maintained an allowlist of scheduled jobs by identifying its top users and content. This allowlist is routinely cleansed and updated for new entries, and schedule owners work together to decide which jobs should stay on it. Due to the automated nature of the cleanup procedure, Trendyol is able to remove duplicate schedules and provide schedule owners the ability to grant permissions for jobs to remain on the allowlist.
- Creating forums where users may work together. Trendyol organised structured programmes that users could participate in and share knowledge with one another, with an emphasis on how high achievers use scheduled tasks. This includes holding frequent meetings, having office hours, and holding conversations in different venues to promote improved teamwork.
- Distributing the times allotted for a job’s performance. In Looker, processing and updating data are among the most computationally demanding operations. Trendyol routinely assesses if jobs that are planned to run during peak hours are actually necessary, and it looks into other options like getting rid of redundant jobs, changing the schedule, or getting rid of outdated and pointless jobs.
- Examining the frequency of work. Determine whether any jobs should be planned for frequent repetition and, if so, think about lowering that rate in order to maximise resource use. For instance, instead of scheduling a job several times a day, the team could determine if scheduling it once a week is adequate to meet user demands.
- Dealing with the schedules of zombies. Trendyol decides if another schedule owner can take over a schedule owner’s assigned tasks in the event that they depart the company. These “zombie” schedules are eliminated from Looker if no other schedule owner can be located.
- Providing cache optimisations and a Hub and Spoke model. Trendyol implemented a hub-and-spoke architecture paradigm, which permits the organisation to implement common business logic while granting specific teams the autonomy to establish their own access control guidelines and policies. Additionally, Trendyol uses Looker’s intelligent caching features to prevent retrieving duplicate data for queries that are identical.
All in all, Trendyol has discovered that offering true cloud governance at scale has been made feasible by combining these initiatives and constantly striving to enhance them. At peak times, Trendyol can now process over 260,000 queries each day, enabling over 1,500 staff members to utilise Looker insights in their everyday work.
