Jekyll2024-01-13T06:41:28+00:00https://skaska.pro/feed.xmlskaskaAI passioned and digital twin amazed cto, solution architect and team leadSergey Kovalev3 Layers of skills in extendable and adaptive LLM-chatbot2024-01-05T19:00:56+00:002024-01-05T19:00:56+00:00https://skaska.pro/3-layers-of-skills<h2 id="introduction">Introduction</h2>
<p>In the dynamic world of artificial intelligence, Language Model (LLM)-powered chatbots stand at the forefront of interactive technology. While foundational skills such as web browsing are commonplace in many chatbots, it is the realms of enhancement and core transformation skills that are truly revolutionizing the capabilities of these digital assistants.</p>
<p><img src="/assets/3-skill-layers/3_layers_of_skills.png" alt="3 skills layers" /></p>
<h2 id="enhancing-skills">Enhancing Skills</h2>
<p>Enhancing the base skill set via a chat interface is now a tangible reality. Innovations like <strong><code class="language-plaintext highlighter-rouge">GptEngineer</code></strong> demonstrate the feasibility of programming through chat-like interactions between the user and the LLM. This layer encompasses more than just crafting new base skills from the ground up; it involves refining and amalgamating existing skills into sophisticated, multifaceted tools. Here, the chat interface serves as a unique alternative to conventional Integrated Development Environments (IDEs), facilitating the creation and modification of code.</p>
<p><img src="/assets/3-skill-layers/enhancement_skills.png" alt="enhancement skills" /></p>
<p>This layer introduces the concept of self-improving skills, initially with user assistance. In a business context, these advanced capabilities are ideally suited for power users and developers.</p>
<h2 id="changing-the-core">Changing the ‘Core’</h2>
<p>Delving deeper, we can modify the foundational interaction layer between the application and the LLM, known as <a href="https://arxiv.org/abs/2305.05364">LLM programs</a>. For instance, consider the <code class="language-plaintext highlighter-rouge">consider_memo_storage</code> method from the AutoGen project:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">consider_memo_storage</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">comment</span><span class="p">):</span>
<span class="s">"""Determines if a user comment should be stored in the database."""</span>
<span class="c1"># Analyzing for a problem-solution context.
</span> <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">analyze</span><span class="p">(</span>
<span class="n">comment</span><span class="p">,</span>
<span class="s">"Does any part of the TEXT ask the agent to perform a task or solve a problem? Answer with just one word, yes or no."</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">if</span> <span class="s">"yes"</span> <span class="ow">in</span> <span class="n">response</span><span class="p">.</span><span class="n">lower</span><span class="p">():</span>
<span class="c1"># Extracting actionable advice.
</span> <span class="n">advice</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">analyze</span><span class="p">(</span>
<span class="n">comment</span><span class="p">,</span>
<span class="s">"Briefly copy any advice from the TEXT that may be useful for a similar but different task in the future. If no advice is present, respond with 'none'."</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">...</span>
</code></pre></div></div>
<p>In this method, the LLM is initially consulted, followed by an interpretation of its response in binary terms (‘yes’/’no’). This is a basic form of an LLM program, involving a call-and-response mechanism. The <code class="language-plaintext highlighter-rouge">consider_memo_storage</code> method, spanning roughly 50 lines, represents well-structured code, yet its operational efficiency and alignment with our objectives remain hypotheses subject to verification.</p>
<p><img src="/assets/3-skill-layers/core_transformation_skill.png" alt="core transformation skill" /></p>
<p>At this layer, we extend our exploration into self-improving LLM applications. By analyzing logs comprising method implementations, inputs, and outputs, we can iteratively refine the LLM program, potentially leveraging <a href="https://arxiv.org/abs/2309.03409">advanced LLMs</a> for optimization.</p>
<h2 id="conclusion">Conclusion</h2>
<p>The exploration of the advanced skill spectrum in LLM-powered chatbots uncovers a world where these tools transcend basic functions. Enhancement skills empower chatbots to grow and adapt through interactive interfaces, stepping beyond the bounds of traditional programming. Core transformation skills take this further, honing the very essence of chatbot functionality. A key consideration emerges: determining which aspects of the application should remain constant throughout its lifecycle, amidst the continuous evolution of its capabilities. This question underscores the intricate balance between innovation and stability in the development of intelligent chatbot applications.</p>Sergey KovalevIntroductionFrom LLM-app to AI/LLM-drive-app2023-11-14T18:00:56+00:002023-11-14T18:00:56+00:00https://skaska.pro/from-llm-app-to-ai-driven-app<h1 id="introduction">Introduction</h1>
<p>In this article, we delve into the evolving landscape of AI-driven applications, particularly focusing on Large Language Models (LLMs). Let’s begin by defining key terms:</p>
<ul>
<li><strong>LLM App</strong>: An application that primarily utilizes Large Language Models for its core functionality and simple scripting for interation with LLM</li>
<li><strong>AI-Driven App</strong>: An alternative approach to AI app development that also actively uses LLMs both for core functionality and control flow</li>
</ul>
<p>Our journey starts with an analysis of David Shapiro’s demonstration of a medical application, specifically the <a href="https://github.com/daveshap/Medical_Intake"><strong>Medical Intake</strong></a> app. This type of application is what we refer to as an LLM-app. Yes, it’s not a production grade app, but a good starting point for our jorney.</p>
<p><img src="/assets/llm-app/intake-app.png" alt="Medical intake app" /></p>
<p>To better grasp this example, we’ll distinguish between LLM interaction logic (or prompt engineering) and control logic (program flow). The Medical Intake app, for instance, uses a straightforward control logic:</p>
<ul>
<li>Engage in a conversation with a patient and condense the chat into <em>notes</em>.</li>
<li>Use these <em>notes</em> to generate multiple documents.</li>
</ul>
<p>Our aim is to demonstrate how integrating domain expertise can enhance the <em>control logic</em>, making the app more flexible and reliable.</p>
<h1 id="steps-to-transition-from-an-initial-llm-app-to-an-ai-driven-app">Steps to Transition from an Initial LLM-App to an AI-Driven App</h1>
<ol>
<li><strong>LLM App</strong>: The basic stage where the app’s functionality is primarily driven by LLM capabilities.</li>
<li><strong>Modularization</strong>: Decompose the intake process into discrete LLM-powered modules. This allows for more structured prompting, response verification, and streamlined testing.</li>
<li><strong>Workflow Implementation</strong>: Incorporating a workflow engine can significantly improve the app by adding monitoring, reliability, and other benefits associated with workflow/business process management systems (WF/BPMS). It also facilitates the integration of new processes.</li>
<li><strong>Evolution to Case Management</strong>: Drawing from extensive experience in BPMS integration, it’s evident that workflows often become more complex, evolving into Case Management Systems. In this scenario, the “case” is the patient intake process, encompassing various stages and actions.</li>
<li><strong>LLM-Driven Decision Making</strong>: The final step involves enabling the LLM to make case management decisions. While individual steps involve LLM interactions, a master LLM is introduced to orchestrate the entire process.</li>
</ol>
<h2 id="llm-app">LLM app</h2>
<p><img src="/assets/llm-app/llm-app.png" alt="LLM app" /></p>
<p>The entire functionality is encapsulated within a single Python script.</p>
<h2 id="modularization">Modularization</h2>
<p><img src="/assets/llm-app/extracted-steps-app.png" alt="Extracted steps app" /></p>
<p>Evolving into a more manageable and testable structure is a logical next step.</p>
<h2 id="workflow">Workflow</h2>
<p><img src="/assets/llm-app/wf-engine-driven-app.png" alt="Worflow app" /></p>
<p>Introducing workflow engines or business process management solutions is the natural progression for step orchestration.</p>
<h2 id="adaptive-case-management">Adaptive case management</h2>
<p><img src="/assets/llm-app/acm-driven-app.png" alt="ACM app" /></p>
<p>Even though the initial process seems simple, the need for greater flexibility becomes apparent over time. Adaptive Case Management (ACM) addresses this by breaking down processes into smaller units, introducing the concept of a case with its lifecycle, and centralizing human expertise in managing these cases.</p>
<h2 id="ai">AI</h2>
<p><img src="/assets/llm-app/ai-driven-app.png" alt="AI app" /></p>
<p>Replacing the human component in ACM with an LLM, we focus on control flow rather than individual tasks. This LLM determines the next steps based on the current state and available actions. The control flow can be enhanced with expert system-style rules, additional information, or alternative data representations. Crucially, integrating human expertise remains vital for moderating AI behavior, adjusting prompts, and introducing new rules. This requires collaboration between AI specialists and domain experts.</p>
<p>Check simplified prompt template as below. Given global task, current state (in form of collected documents) and available actions LLM have to come up with next step to perform.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{mission}
{goal}
Currently you are in the middle of the {process_name} process.
Below is list of activities you can use to complete the goal and already collected documents from the patient.
===
Activities
{activities}
===
Documents available
{documents}
===
Common happy path is to execute activities in the following order:
{happy_path}
You are free to choose any activities that you think will help you to complete the goal.
Provide name of the activity you want to execute and list of parameters to execute it.
</code></pre></div></div>
<p>This approach to control the flow can further be extented with additional rules (expert system style approach) and other information or other ways to represent same information (i.e. log => actions taken can be presented as results of these actions or rather action description). Additinal flexibility can and should be added by including human expert to moderate AI behavoir by chanaging prompts, introducing new rules etc. Actually there should be 2 experts - one is more AI related with prompt engineering experience, and the second is domain expert ()</p>
<h1 id="ai-driven-app-architecture">AI-Driven App Architecture</h1>
<p>Key components of an AI-Driven App:</p>
<ul>
<li><strong>Core LLM</strong>: Retains the same LLM used in the initial app, now augmented with verification mechanisms and structured prompts to improve step reliability.</li>
<li><strong>Step Types</strong>: Introduces a variety of steps, including expert consultations and data retrieval from external systems.</li>
<li><strong>LLM for Step Management</strong>: A new LLM layer is implemented to determine the sequence of steps, starting as a simple expert system and evolving with the application.</li>
<li><strong>Hardcoded Framework</strong>: A foundational structure that encapsulates the concept of steps/actions and their interaction with the executor-LLM.</li>
<li><strong>Rules</strong>: Knowledge segments that enhance AI performance in complex scenarios.</li>
<li><strong>Structured Document Storage</strong>: An alternative to raw vector storage, facilitating better data management.</li>
</ul>
<p><img src="/assets/llm-app/arhi_blocks.png" alt="archi blocks" /></p>
<h1 id="conclusion">Conclusion</h1>
<p>As Demis Hassabis once remarked, delegating more of the initial task to neural networks leads to superior outcomes. However, not everyone has access to high-end computational resources like the <em>GH200</em> clusters. Therefore, understanding and exploring hybrid methodologies is crucial in developing efficient and effective AI.</p>Sergey KovalevIntroductionWorkforce agent2023-10-25T18:00:56+00:002023-10-25T18:00:56+00:00https://skaska.pro/workforce-agent<p><strong>Consider</strong> this as part 3 of the “Sequential Planner” series: <a href="/problems-of-common-sequential-planners">part 1</a>, <a href="/sequential-planner-v2">part 2</a>.</p>
<h2 id="idea">Idea</h2>
<p>Imagine having unlimited access to <strong>remote</strong> mid and junior-level knowledge workers. The only limitation? They don’t use Zoom. However:</p>
<ul>
<li>They are proficient with Jira and Confluence.</li>
<li>They are available 24/7.</li>
<li>They meticulously track their work and validate outcomes.</li>
<li>They seek assistance when needed.</li>
<li>… and much more.</li>
</ul>
<h3 id="interaction-scenario">Interaction Scenario</h3>
<p>Here’s an illustrative interaction scenario:</p>
<ul>
<li>A user converses with a <strong>manager agent</strong> (MA) to define a task.</li>
<li>The MA logs the task in Jira and assigns it to the appropriate <strong>worker agent</strong> (WA).</li>
<li>The WA drafts a strategy to accomplish the task, seeking clarifications or supplementary documents if necessary. This plan might manifest as sub-tasks within Jira.</li>
<li>Once the plan is set, the WA commences its execution, continually updating the status of each sub-task.</li>
<li>After completing all sub-tasks, the MA conducts a verification/validation. If all criteria are met, the status of the primary task is marked as <em>done</em>.</li>
</ul>
<h2 id="current-state-of-agent-development">Current State of Agent Development</h2>
<h3 id="simple">Simple</h3>
<p>Agents are often perceived as mere remote procedure calls rather than entities capable of planning and execution. Sometimes, they’re even externally orchestrated (e.g., check <strong>agentprotocol</strong>).</p>
<h3 id="planners">Planners</h3>
<p>To grasp the limitations of current planners and potential improvements, refer to <a href="/problems-of-common-sequential-planners">part 1</a> and <a href="/sequential-planner-v2">part 2</a>.</p>
<h3 id="reasoning">Reasoning</h3>
<p>CoT, ToT, GoT – these methods focus predominantly on reasoning, not on collaborative task resolution. Nonetheless, they can certainly serve as <em>low-level</em> approaches.</p>
<h2 id="main-hint-workflow-engines-analogy">Main Hint: Workflow Engines Analogy</h2>
<p>Indeed, workflows are repetitive processes. However, they offer a prime analogy for understanding task execution. Explore platforms like Camunda, Nintex, Azure Logic Apps, etc. Envision task setup akin to configuring a workflow, accounting for:</p>
<ul>
<li>Internal variables.</li>
<li>Sub-processes.</li>
<li>Updates to external systems.</li>
<li>And more.
The task execution can be likened to running such a workflow – but just once.</li>
</ul>
<h2 id="missing-ingredients">Missing Ingredients</h2>
<p>There are four pivotal components currently absent in state-of-the-art agent developments:</p>
<ul>
<li>An innovative task planning methodology, incorporating hierarchical task decomposition and the ask-plan-execute strategy.</li>
<li>Flexibility concerning multiple kernels/LLMS and intricate internal data types (entities, collections).</li>
<li>Resilience regarding external task and sub-task states and statuses.</li>
<li>Seamless integration with task management systems, wikis, and document portals.</li>
</ul>
<p><img src="/assets/sequential-planners/new_core.png" alt="New Agent Core" /></p>
<h2 id="agent-core">Agent Core</h2>
<h3 id="three-modules">Three Modules</h3>
<p>The core functionality of an agent can be divided into three modules:</p>
<ul>
<li><strong>Manager</strong>:
<ul>
<li>Engages with users or high-level systems.</li>
<li>Primarily aims to draft a detailed task description.</li>
</ul>
</li>
<li><strong>Planner</strong>:
<ul>
<li>Utilizes the detailed task description to devise an execution plan.</li>
<li>Can seek further information from the manager or report issues if the task plan is unfeasible due to insufficient skills or data.</li>
<li>Delivers a search tree alongside the execution plan, offering insights to the executor in case of execution issues.</li>
<li>The plan is uploaded to the task management system (TMS).</li>
</ul>
</li>
<li><strong>Executor</strong>:
<ul>
<li>Carries out the plan either autonomously or with the assistance of <em>simple agents</em>.</li>
<li>Updates task statuses in the TMS during execution. Some artifacts may also be generated in wikis, portals, or code repositories.</li>
</ul>
</li>
</ul>
<h3 id="planning-execution">Planning Execution</h3>
<p>Approach the agent as you would a junior developer: don’t anticipate exhaustive domain knowledge or flawless execution. Allow the agent to outline its approach to achieving the desired outcome (see <strong>Hierarchical task split</strong>). A hierarchical planner can either produce an executable plan or one of two errors: lack of functions or lack of information. These inadequacies can be interpreted as calls for assistance.</p>
<h3 id="tasks-wikis-etc">Tasks, Wikis, etc.</h3>
<p>While the conventional approach (except for MemGPT) retains the entire execution log in context, the proposition here is to use a wiki and/or task description to house intermediary data. The motivations are:</p>
<ul>
<li>It’s pivotal for parallel execution and multiple agents working on sub-tasks.</li>
<li>Agent activities are easily traceable, thanks to standard task management solutions and wikis.</li>
</ul>
<p>For this to be feasible, agents must understand tasks, sub-tasks, their interrelations, statuses, etc., as well as the structure of the wiki. This understanding is essential as different teams manage projects in varied ways.</p>
<h2 id="future-work">Future Work</h2>
<p>The way forward is clear: develop the <strong>Core</strong> for Workforce Agents.</p>Sergey KovalevConsider this as part 3 of the “Sequential Planner” series: part 1, part 2.Sequential planner v22023-10-25T05:00:56+00:002023-10-25T05:00:56+00:00https://skaska.pro/sequential-planner-v2<p>This is part 2 (<a href="/problems-of-common-sequential-planners">part 1</a>).</p>
<h2 id="initial-idea-for-plan-generation">Initial Idea for Plan Generation</h2>
<p>Create a planner that can:</p>
<ul>
<li>Confirm that the final result structure and acceptance criteria are met.</li>
<li>Respond to the calling system if some information or functionality is missing.</li>
<li>Use a top-to-bottom approach and split tasks into smaller ones if necessary.</li>
<li>Work with collections/lists and other data structures.</li>
</ul>
<h2 id="implementation">Implementation</h2>
<p>Main points:</p>
<ul>
<li>The user or invoking system provides information necessary to formulate an extended version.</li>
<li>The planner consists of two modules: a task manager that interacts with the user/system and a planner that creates a plan and interacts with the task manager.</li>
</ul>
<p><img src="/assets/sequential-planners/planner_v2.png" alt="Sequential Planner v2" /></p>
<h3 id="extending-task-description">Extending Task Description</h3>
<p>The task description that is passed to the planner needs to have not only the task text but also:</p>
<ul>
<li>High-level values for the planner (i.e., mission, project context).</li>
<li>Input information description.</li>
<li>Resulting documents description.</li>
<li>Acceptance criteria.</li>
</ul>
<p>This extended description can be either the result of interaction with the user/system or generated from project description and documentation, common sense, etc.</p>
<h3 id="hierarchical-task-splitting">Hierarchical Task Splitting</h3>
<p>The start of the algorithm is almost the same as for a conventional sequential planner. The difference is the extended task description, but we can ignore it for now.</p>
<p><strong>Given</strong>:</p>
<ul>
<li>Task as a string.</li>
<li>Extended task description.</li>
<li>List of available functions in the form the planner can understand.</li>
</ul>
<p><strong>Algorithm</strong>:</p>
<ul>
<li>Try to solve the task using a conventional planner and existing functions. If solved => <strong>hurrah! Done</strong>.</li>
<li>Ask the planner to split the task into several (not many 5-10) stages with a detailed description (same as the extended description for the initial task) on each step.</li>
<li>[Recursion]
<ul>
<li>For each stage, try to resolve it by a conventional planner or by splitting it into smaller stages.</li>
<li>If recursion works => fine!</li>
<li>If not, dig down till some level.</li>
</ul>
</li>
</ul>
<p><strong>Key points</strong>:</p>
<ul>
<li>Keep records of all explored steps (tree/graph of hypotheses).</li>
<li>Be creative and try splitting into stages multiple times (i.e., high <strong>temperature</strong>).</li>
<li>Distinguish between two problems for a solution not found (stage cannot be resolved with conventional planners):
<ul>
<li>Cannot apply conventional planner due to a lack of functions.</li>
<li>Additional information/document is required.</li>
</ul>
</li>
</ul>
<p>The main difference of the described approach from <strong>[X]_of_Thoughts</strong> approaches (CoT, ToT, GoT, etc.) is that we are okay to introduce non-defined steps and define them later, instead of doing a search only in the space of available functions.</p>
<p><img src="/assets/sequential-planners/hierarchical_task_split.png" alt="simplified version of hierarchical task split algorithm" /></p>
<h3 id="example-a-simple-case--success-plan">Example A: Simple Case => Success Plan</h3>
<p>Below is a diagram describing a simple case that is identical to a common sequential planner:</p>
<ul>
<li>The task is successfully split into several activities/subtasks.</li>
<li>Each activity is mapped to an existing skill from the given skillset.</li>
<li>The input document is mapped to the input of <em>activity A</em> (skill A).</li>
<li>The output document is the output from the last <em>activity C</em> (skill C).</li>
</ul>
<p><img src="/assets/sequential-planners/simple_case_plan.png" alt="Simple Case Plan" /></p>
<h3 id="example-b-simple-case--problematic-plan">Example B: Simple Case => Problematic Plan</h3>
<p>This is a more complex case:</p>
<ul>
<li>The planner was unable to map the initial task into a sequence of skills.</li>
<li>The planner split the initial task into high-level activities.
<ul>
<li>Activity #1 and activity #2 were successfully mapped into a sequence of skills.</li>
<li>But there was a problem with activity #3 - it required additional information (document X).</li>
</ul>
</li>
<li>Though the planner tried to create a plan several times, <strong>document X</strong> was always required.</li>
<li>This request for <strong>document X</strong> was passed to the <strong>task manager</strong> module.</li>
</ul>
<p><img src="/assets/sequential-planners/multilevel_case_plan.png" alt="Bad Planning" /></p>
<h2 id="conclusion">Conclusion</h2>
<p>A high-level description of the algorithm for advanced planning was given, and initial ideas were implemented.</p>Sergey KovalevThis is part 2 (part 1).Sequential planner flaws2023-10-24T05:00:56+00:002023-10-24T05:00:56+00:00https://skaska.pro/problems-of-common-sequential-planners<blockquote>
<p><strong>Disclaimer #1:</strong> The insights shared in this article are based on experiments conducted with Semantic Kernel (version < 1.0). While Langchain exhibits similar challenges, I have not delved deeply into its workings.</p>
</blockquote>
<blockquote>
<p><strong>Disclaimer #2:</strong> The crux of this discussion revolves around devising a plan, rather than its execution.</p>
</blockquote>
<blockquote>
<p><strong>Disclaimer #3:</strong> Concepts such as CoT, ToT, GoT, etc., are not pertinent to this discussion.</p>
</blockquote>
<h2 id="the-sequential-planner-flow">The Sequential Planner Flow</h2>
<p>An overview of the current implementation of the sequential planner:</p>
<p><strong>Given:</strong></p>
<ul>
<li>A goal expressed as a string.</li>
<li>A list of available functions that the planner can interpret (SK => plugins).</li>
</ul>
<p><strong>Flow:</strong></p>
<ul>
<li>Extract a concise list of functions that may be relevant for the task at hand.</li>
<li>Formulate a prompt encompassing:
<ul>
<li>Descriptions of the selected functions.</li>
<li>Directions to craft a plan.</li>
<li>The specified goal.</li>
</ul>
</li>
<li>Engage the LLM with the crafted prompt to obtain the plan.</li>
</ul>
<h4 id="plan-example">Plan Example:</h4>
<p>Given the objective: “Summarize an input, translate it to French, and e-mail it to John Doe”, the following plan was devised:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Steps:
- SummarizePlugin.Summarize input='$INPUT' => SUMMARY
- WriterPlugin.Translate input='$SUMMARY' => TRANSLATED_SUMMARY
- email.GetEmailAddress input='John Doe' => EMAIL_ADDRESS
- email.SendEmail input='$TRANSLATED_SUMMARY' email_address='$EMAIL_ADDRESS'
</code></pre></div></div>
<h2 id="limitations-of-the-current-approach">Limitations of the Current Approach</h2>
<p>While this methodology suffices for rudimentary tasks with concise plans, it falters when addressing more intricate challenges. Some of the pitfalls include:</p>
<ul>
<li>Irrespective of the prompt instructions, the generated plan may inadvertently employ collections as variables.</li>
<li>The structure of the final solution remains ambiguous. Merely augmenting instructions is ineffective, given the absence of a verification mechanism.</li>
<li>Should a function crucial for achieving the goal be absent, the invoking system remains oblivious.</li>
<li>Similarly, if supplementary information is essential for goal accomplishment, the system remains uninformed.</li>
</ul>
<h2 id="a-glimmer-of-hope">A Glimmer of Hope</h2>
<p>Firstly, LLMs at the caliber of GPT-4 are equipped to devise algorithms to tackle almost any challenge (e.g., the 12 tasks delineated in the <a href="https://evals.alignment.org/blog/2023-08-01-new-report/">ARC report</a>). This includes the ability to decompose a task into more manageable sub-tasks. While the resultant algorithm may not always be optimal, leveraging multiple generations (with a non-zero temperature) could pave the way for satisfactory outcomes.</p>
<h2 id="proposed-solution">Proposed Solution</h2>
<p>The aspiration is to develop a planner capable of:</p>
<ul>
<li>Ensuring that the final result adheres to the defined structure and meets acceptance criteria.</li>
<li>Informing the invoking system in case certain information or functionality is lacking.</li>
<li>Employing a top-down strategy, further dissecting tasks as needed.</li>
<li>Seamlessly integrating with collections/lists and other data structures.</li>
</ul>Sergey KovalevDisclaimer #1: The insights shared in this article are based on experiments conducted with Semantic Kernel (version < 1.0). While Langchain exhibits similar challenges, I have not delved deeply into its workings.AI is not ‘slow kind of country’2023-10-17T05:00:56+00:002023-10-17T05:00:56+00:00https://skaska.pro/ai-not-slow-kind-of-country<p>Back in business - after dramatic pause and 3 projects behind 😀
Below is compilation of 3 small posts from <a href="https://www.linkedin.com/in/skovalev/recent-activity/all/">LinkedIn</a></p>
<h3 id="context">Context</h3>
<ul>
<li>Red queen race</li>
<li>AI adoption</li>
<li>LLM is junior</li>
</ul>
<h2 id="the-red-queens-ai-race">The Red Queen’s AI Race</h2>
<p>In the realm of tech, AI adoption feels reminiscent of the Red Queen’s race from “Alice in Wonderland” - indeed, we’re no longer in ‘a slow sort of country!’</p>
<p><img src="/assets/ai-transformation/red_queen_race.jpeg" alt="red queen race - AI plans" /></p>
<p><strong>Participate</strong></p>
<ul>
<li>Every individual in an organization must now harness the capabilities of AI tools and integrate them into their daily tasks.</li>
<li>In the upcoming month, the focus should shift to crafting in-house AI solutions to refine and elevate processes.</li>
<li>And by the next quarter, it’s time to weave AI seamlessly into product lines, take the helm of industry-wide AI initiatives, and seek inspiration from platforms like ChatGPT for a fresh wave of ideas!😀</li>
</ul>
<h2 id="embracing-ai">Embracing AI</h2>
<p>Many of us have dabbled with tools like ChatGPT, StableDiffusion, and other AI utilities. However, integrating AI at an enterprise or departmental level or weaving it into an application remains a challenging feat. The absence of a one-size-fits-all AI framework does complicate matters. Currently, AI projects often feel more akin to exploratory research than traditional software development. Presented here is a methodical approach to ease AI adoption. It’s not a rigid framework, but a flexible strategy to consider.</p>
<p><img src="/assets/ai-transformation/ai_adoption.jpeg" alt="AI Adoption Incrementally" /></p>
<p><strong>The AI Adoption Approach</strong>:</p>
<ol>
<li><strong>Identify Potential</strong>: Draft a list of use cases that can benefit from AI acceleration or enablement.</li>
<li><strong>Three Pillars of Focus</strong>: For each use case, center your strategy around three primary pillars: data, algorithm, and user experience (UX).
<ul>
<li><strong>Data</strong>: This encompasses the information that will be employed to train or underpin the AI, establishing it as a strategic asset.</li>
<li><strong>Algorithm</strong>: Instead of chasing the latest AI innovations (given the inundated state of AI news feeds), select a reliable, time-tested algorithm.</li>
<li><strong>UX</strong>: Develop AI-backed widgets for integration into your portal or application. Always have an administrative interface or dashboard in place to monitor performance and gauge user engagement.</li>
</ul>
</li>
<li><strong>Quality over Quickness</strong>: Prior to rolling out any solution (even to early adopters), it’s crucial to vet its reliability. Due to the unpredictable nature of AI, it’s advisable to employ a “red team” or an independent group to test and challenge the system’s security and reliability.</li>
</ol>
<p><strong>The Key</strong>: Begin with simplicity. A common mistake is to invest all your resources into perfecting one pillar at the outset. Such an approach often stalls projects. Instead, view each pillar as a container that you gradually fill over successive development cycles. For the first cycle, pour a modest amount into each container. Once you have a functioning system in place - which is often the hardest part organizationally - continue to incrementally enhance each pillar over time.</p>
<h2 id="llm-is-junior">LLM is junior</h2>
<p>While general large language models (LLM) like Claude2, Bard, ChatGPT, and others are incredibly versatile, it’s essential to recognize that their performance on niche domain-specific tasks may not always match an expert’s acumen. Instead of expecting senior-level expertise, it might be more realistic to anticipate competence equivalent to a junior or mid-level professional. Strategies to maximize LLM effectiveness include:</p>
<ul>
<li>Simplifying tasks or dividing them into more manageable sub-tasks.</li>
<li>Giving illustrative examples for better understanding.</li>
<li>Actively ensuring the results’ accuracy.</li>
</ul>
<p><img src="/assets/ai-transformation/llm_is_junior.jpeg" alt="llm is junior" /></p>
<p><strong>Verification Techniques:</strong></p>
<ul>
<li>Integrate a verification step within your prompt. If the model’s response doesn’t satisfy the verification criteria, request reconsideration.</li>
<li>Design the LLM interaction to include multiple steps or loops for self-checks and validation.</li>
<li>Engage a secondary model or system with a distinct prompt to corroborate the initial LLM’s responses.</li>
</ul>
<h2 id="book-is-book-not-a-knowledge">Book is book, not a knowledge</h2>
<p><a href="https://en.wikipedia.org/wiki/DIKW_pyramid">DIKW pyramid</a></p>
<blockquote>
<p>The DIKW pyramid, also known variously as the DIKW hierarchy, wisdom hierarchy, knowledge hierarchy, information hierarchy, information pyramid, and the data pyramid, refers loosely to a class of models for representing purported structural and/or functional relationships between data, information, knowledge, and wisdom. “Typically information is defined in terms of data, knowledge in terms of information, and wisdom in terms of knowledge”.The DIKW acronym has worked into the rotation from knowledge management. It demonstrates how the deep understanding of the subject emerges, passing through four qualitative stages: “D” – data, “I” – information, “K” – knowledge and “W” – wisdom</p>
</blockquote>
<p><img src="/assets/ai-transformation/book_is_a_book.png" alt="book is a book" /></p>
<p>In the world of Knowledge Management, let’s break down the differences between data, information, and knowledge in a more casual way. A common misunderstanding these days is thinking that just tossing documents into a digital storage bin and using fancy Retrieval Augmented Generation (RAG) with Large Language Models (LLM) will magically transform your system into a Knowledge Management powerhouse. Nope, it doesn’t work that way – you’re still squarely in the realm of Information Management.</p>
<p>To put it simply, when it comes to knowledge, think of it as <strong>“active”</strong> compared to information, which is more <strong>“passive”</strong>.
Now, let’s expand on this with some relatable examples:</p>
<p><strong>Knowledge Gets Things Done:</strong>
Knowledge isn’t just knowing stuff; it’s about having the know-how to get things done. Imagine you have a text with an algorithm that you understand and can execute. That text becomes knowledge because you can put it into action. You’re like a problem-solving wizard with that knowledge!</p>
<p><strong>Information is Like Clues:</strong>
But, if the same text lands in front of someone who doesn’t have the magic skills to run that algorithm, it’s just information. It’s like giving them a bunch of clues, but they don’t quite know how to use them.</p>
<p><strong>Data is the Raw Material:</strong>
Now, picture this: someone who not only can’t execute the algorithm but also doesn’t understand the language of that text. For them, it’s not even information; it’s just raw data. It’s like staring at a jumble of characters without a clue.
The point here is that in Knowledge Management, we’re not just about collecting heaps of data or storing loads of information. It’s about making sure people can take that info and do something useful with it. True Knowledge Management is about turning data into actionable information and eventually into practical know-how that drives results. It’s like having the right tools in your toolbox – it’s not about how many you have, but how well you can use them to build something amazing.</p>Sergey KovalevBack in business - after dramatic pause and 3 projects behind 😀 Below is compilation of 3 small posts from LinkedInSome thoughts on Architecture Decisions2021-12-20T05:45:56+00:002021-12-20T05:45:56+00:00https://skaska.pro/architecture-decisions-thoughts<h1 id="what-is-it">What is it</h1>
<p>An architecture decision (AD) is a software design choice that addresses a significant requirement.
From <a href="https://github.com/joelparkerhenderson/architecture-decision-record#what-is-an-architecture-decision-record">here</a></p>
<p>During project team makes multiple ADs.
Each decision is made at some point in time and within specific context - project history, current functional requirements (i.e. set of user stories), set of non-functional requirements, product vision, clients in pipe etc. This context is subject to change in time.</p>
<h1 id="track-it">Track it</h1>
<p><a href="https://github.com/joelparkerhenderson/architecture-decision-record">ADR</a> is common approach to track AD</p>
<p>Very good example of ADR usage - <em>A Case Study - Konstantin Kudryashov - DDD Europe 2020</em></p>
<!-- Courtesy of embedresponsively.com -->
<div class="responsive-video-container">
<iframe src="https://www.youtube-nocookie.com/embed/x5YmBevdjVg" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
</div>
<h1 id="evaluate-it">Evaluate it</h1>
<p>What does it mean to evaluate AD? When should it be done?
<img src="assets/architecture-decisions/architecture-decisions-process.png" alt="" /></p>
<h1 id="quadrants">Quadrants</h1>
<p><img src="assets/architecture-decisions/architecture-decisions-quadrant.png" alt="architecture decisions - effort vs business value" /></p>
<h2 id="magic">Magic</h2>
<p>The art of gathering low hanging fruits.
Bringing in some new app, lib, pattern that proved to work in similar domains, but it’s unfamiliar to project team.
As an example check article <a href="/not-only-sql">Sql is not everything you need</a></p>
<h2 id="safe-play">Safe play</h2>
<p><strong>No pain no gain</strong> or <strong>agile trap</strong> quadrant.
How is it <em>an agile trap</em>?
All ADs that are made with only looking at current needs of the project. Is it bad – no. But this kind of decision making фззкщфср will probably make you start <em>v2.0</em> of your app much sooner than expected. Reason - no <strong>platform vision</strong> in architecture.</p>
<h2 id="vision">Vision</h2>
<p>Hardest one, because it’s easy to fail into <em>overengineering</em>.</p>
<h2 id="overengineering">Overengineering</h2>
<p>Nothing is wrong with this quadrant, unless all decisions end up here.</p>
<h1 id="multiple-ads-not-one">Multiple ADs, not ONE</h1>
<p><img src="assets/architecture-decisions/architecture-decisions-portfolio.png" alt="multiple architecture decisions" /></p>
<h1 id="value-over-time">Value over time</h1>
<p>Good example is AD from <em>vision</em> quadrant. Rarely you can put all ADs right in that quadrant from the very beginning. Initially these decisions tend to move to <em>overengineering</em> quadrant first, and if team is lucky and persistent, it will eventually move to <em>vision</em>.</p>
<p><img src="assets/architecture-decisions/architecture-decisions-value-evolution-over-time.png" alt="architecture decisions value evolution over time" /></p>
<h1 id="strategy">Strategy</h1>
<p>Let’s recap</p>
<ul>
<li>Safe play is obvious choice for most companies. But that’s mid-term trap.</li>
<li>To get some magic – networking is an answer, or bringing in some external people with different domain expertise.</li>
<li>Vision – requires both technical and product components.</li>
</ul>
<p>Team should work with ADs as with any portfolio using <strong>risk-reward</strong> estimation approach, i.e. in general, accept some risk of <strong>overengineering</strong> but manage it.</p>
<p>Managing risk of <em>overengineering</em> means</p>
<ul>
<li>mistakes will happen, but they are allowed</li>
<li>reevaluate AD often</li>
<li>be transparent and honest with team</li>
</ul>
<h1 id="guesstimate">Guesstimate</h1>
<p>Do not underestimate element of luck in decision making - team operates with incomplete information in fast changing world.</p>Sergey KovalevWhat is it An architecture decision (AD) is a software design choice that addresses a significant requirement. From hereSQL DB is not ‘the only thing you need’2021-02-14T04:45:56+00:002021-02-14T04:45:56+00:00https://skaska.pro/not-only-sql<h1 id="introduction">Introduction</h1>
<p>After facing some legacy systems lately and complains ‘sql db is bottleneck’ and ‘we have huge sql cluster’, usually accompanying with something like ‘we plan to switch to microservices’. Idea came up to my mind that somehow ‘microservices pattern’ is usually interpreted as separating ‘calculation’ or ’processing’ but not about data storage separation.</p>
<p>SQL database in legacy solution is usually used for storing and sometime processing of</p>
<ul>
<li>Business data</li>
<li>‘Stateful activity’</li>
<li>Event logs (business, IoT, system)</li>
<li>Business logic execution</li>
</ul>
<p><img src="/assets/sql-spaghetti/usage_scenarios.png" alt="sql db usage scenarios" /></p>
<p>Let’s dive deeper into each scenario</p>
<h1 id="business-domain-data">Business domain data</h1>
<p>Business data – even though you can move it to some document storage that make sense sometime, just leave it there for now. There are usually better options to start optimization with.
</p>
<h1 id="stateful-activities">Stateful activities</h1>
<p>Any activity that will end up one day but needs some persistence till that happy moment. Usually it’s huge piece of work to do it in proper way but it looks pretty small until you dive deeper. Good new there are already solutions to handle all the heavy load.</p>
<ul>
<li>User interaction requests. When system needs some input from user. Common cases for enterprise solutions are all sorts of business process automation tasks (‘approve document’, ‘fulfill form’ etc.).</li>
<li>Workflows (‘saga pattern’)</li>
<li>Message queuing</li>
<li>ETL artifacts. Some data is processed on regular basis and process is usually multistage, so intermediary results are stored in tables.</li>
</ul>
<p><img src="/assets/sql-spaghetti/stateful_activity.png" alt="event logs" /></p>
<h2 id="common-symptoms">Common symptoms</h2>
<ul>
<li>table that is constantly updating (number of <strong>update</strong> operations is significant) and ‘loosely coupled’ with other tables in DB</li>
<li>table with ‘status’/‘processed’ field or ‘task’ in table name</li>
</ul>
<table>
<thead>
<tr>
<th>Scenario</th>
<th>Symptoms</th>
<th>Solutions</th>
<th>App examples</th>
</tr>
</thead>
<tbody>
<tr>
<td>User interaction</td>
<td>tables refer to ‘user data’ table and in permanent <em>update</em> process</td>
<td>BPMS</td>
<td>Camunda</td>
</tr>
<tr>
<td>WF, sagas</td>
<td>check for <em>common symptoms</em></td>
<td>Specialized WF solutions</td>
<td>Temporal, Cadence, Netflix Conductor</td>
</tr>
<tr>
<td>Queues</td>
<td>Look at solution code – one or several ‘workers’ is the key.</td>
<td>AMQP, Message bus</td>
<td>RabbitMq, Azure MessageBus</td>
</tr>
<tr>
<td>ETL</td>
<td>Same as queues (look for ‘workers’ in the code) but tables usually contains bigger records</td>
<td>Build datapipe or use actor approach for ‘stateful serverless’</td>
<td>Airflow, Luigi vs AKKA, Orleans</td>
</tr>
</tbody>
</table>
<p> </p>
<h1 id="event-logs">Event logs</h1>
<p>Journals of ‘actual events’ that are not always well structured.</p>
<ul>
<li>IoT/IIoT data. User interacts with some hardware and you log this event or some device’s sensor sends data system – think about it as IoT-data-streams and handle these events in ‘modern way’</li>
<li>Business events. External system is generates data that needs to be processed</li>
<li>System, application, solution events, user UI interaction events etc.</li>
</ul>
<p><img src="/assets/sql-spaghetti/log.png" alt="event logs" /></p>
<h2 id="common-symptoms-1">Common symptoms</h2>
<ul>
<li>huge (in terms of number of rows) tables that constantly growing with no links to other tables</li>
<li>‘append only’ tables that periodically cleaned up (mostly manually)</li>
</ul>
<table>
<thead>
<tr>
<th>Scenario</th>
<th>Solution</th>
<th>App examples</th>
</tr>
</thead>
<tbody>
<tr>
<td>IoT</td>
<td>*simple processing</td>
<td>MongoDb, Kafka, Clickhouse</td>
</tr>
<tr>
<td>Business events</td>
<td>*simple processing</td>
<td>MongoDb, MessageBus, Data Warehouse</td>
</tr>
<tr>
<td>System events</td>
<td>Logging solutions</td>
<td>ELK, Prometheus</td>
</tr>
</tbody>
</table>
<h4 id="simple-processing"><em>*Simple processing</em></h4>
<ul>
<li>Store raw incoming events in NoSql Db (MongoDB) with TTL</li>
<li>Create data-pipe to process them (Airflow, Luigi, Spark) or go with RabbitMQ and console-apps</li>
<li>Save consumable results in Data Warehouse or OLAP-friendly DB (Clickhouse)</li>
</ul>
<p><img src="/assets/sql-spaghetti/simple_pipeline.png" alt="simple processing pipeline" /></p>
<p>Check ‘stream processing’ approaches, but it might be too much at this stage.</p>
<h1 id="stored-procedures-and-triggers">Stored procedures and triggers</h1>
<p>Last but not the least. You have tons of stored procedures and triggers, in other words lots of solutions business logic leaves there</p>
<p><img src="/assets/sql-spaghetti/stored_procedures.png" alt="simple processing pipeline" /></p>
<table>
<thead>
<tr>
<th>Component</th>
<th>Example</th>
<th>Solution</th>
</tr>
</thead>
<tbody>
<tr>
<td>Triggers</td>
<td>Start ‘next step’ of processing after ‘status’ is changed</td>
<td>check ‘stateful activities’ part of this article</td>
</tr>
<tr>
<td>Heavy aggregating procedures</td>
<td>Generating reports or spreadsheets for export</td>
<td>check datapipes</td>
</tr>
<tr>
<td>Scheduled heavy updating procedures</td>
<td>Monthly salary payments in big enterprise</td>
<td>BPMS/WF and/or immutable data pattern</td>
</tr>
</tbody>
</table>
<p><strong>(Very opinionated)</strong> usually lot’s stored procedures and/or triggers is pure evil.</p>
<h1 id="conclusion">Conclusion</h1>
<p>There are definitely more cases of using SQL DBs and other persistence approaches than mentioned in this short article.
Nothing wrong to use DB for all the things mentioned above, but there are special solutions to handle these tasks with</p>
<ul>
<li>More functionality. Even if don’t need it now, it’s like a good old hammer – you’ll see many nails around.</li>
<li>Better tested, production ready and better optimized</li>
<li>Community to support and developers to find</li>
</ul>Sergey KovalevIntroduction After facing some legacy systems lately and complains ‘sql db is bottleneck’ and ‘we have huge sql cluster’, usually accompanying with something like ‘we plan to switch to microservices’. Idea came up to my mind that somehow ‘microservices pattern’ is usually interpreted as separating ‘calculation’ or ’processing’ but not about data storage separation.Digital Twins: Implementation2020-11-11T02:00:00+00:002020-11-11T02:00:00+00:00https://skaska.pro/digital-twins-details<h1 id="digital-twins-implementation">Digital twins: implementation</h1>
<p>Let us dive deeper into Digital Twins implementation. The way I see 3 big classes of Digital Twins is described <a href="digital-twins-3-types">here</a>.
First, we will look at types of basic building blocks we need to implement Digital Twins in IIoT solution</p>
<ul>
<li>digital twin instance implementation extension</li>
<li>management extension for digital twin instances</li>
<li>data flows</li>
</ul>
<p>Then we take a look at 2 projects and how these blocks can help solve real life scenarios.</p>
<h2 id="base-digital-twins">Base Digital Twins</h2>
<p>Goal is to <em>extend</em> (adopt?) model for simpler yet more powerful tool to model digital twins.</p>
<p><img src="assets\dt-implemetation\diti base.png" alt="digital twin implementation diagram" /></p>
<p>Types of blocks</p>
<ol>
<li>Devices/sensor level
<ul>
<li>Camera, ble/uwb anchor etc</li>
<li>Simple (on/off) or more complicated state-machine</li>
<li>Sending live state data every 1 second/minute etc</li>
</ul>
</li>
<li>Algorithmic Estimator
<ul>
<li>Classical deterministic or neural nets algorithm to estimate internal state of the object or process</li>
</ul>
</li>
<li>Probabilistic estimator
<ul>
<li>For example estimating BLE asset tag</li>
<li>Telemetry as input</li>
<li>Particle filter</li>
<li>Keeps prob-distribution as a internal state</li>
<li>Outputs – best/mean/distribution</li>
</ul>
</li>
<li>Aggregator
<ul>
<li>Combine 2 or more streams of potentially different frequency and latency into one</li>
</ul>
</li>
<li>System estimator
<ul>
<li>Inputs are from estimators</li>
<li>Conflict re-solving</li>
</ul>
</li>
</ol>
<h2 id="management-extension">Management extension</h2>
<p>Management block (or extension) is used when we have more than one DT of the same type and one data flow that needs to be separated into several (one for each DT) according to some rule.</p>
<p><img src="assets\dt-implemetation\diti manager.png" alt="digital twim manager implementation diagram" /></p>
<h2 id="data-flows">Data Flows</h2>
<p>We have 4 data flow types</p>
<ol>
<li>Telemetry from IoT devices</li>
<li>State and State Estimations from Digital Twins</li>
<li>Events (like errors)</li>
<li>Configuration and sync business data</li>
</ol>
<h2 id="example-1-rtls">Example #1. RTLS</h2>
<ul>
<li>Remote site with ‘thin connection’ to cloud</li>
<li>RTLS BLE-based infrastructure that produce ~10Mb sec IIoT data streams</li>
<li>On-site RTLS server that estimates location of BLE tags</li>
<li>Smartphones are part of solution and produce 2 data flows – real-time BLE-tag info and periodical GPS based location</li>
<li>In cloud all data is aggregated using information from MES that provides employee-smartphone relation</li>
</ul>
<p>Simplified schema of main blocks of RTLS solution</p>
<p><img src="assets\dt-implemetation\tayshet rtls architecture.png" alt="RTLS solution architecture" /></p>
<h3 id="dt-managers">DT Managers</h3>
<p>Digital Twins part that contains</p>
<ul>
<li>DT manager for BLE-Tag Twin</li>
<li>DT-manager for Employee Twin</li>
</ul>
<p><img src="assets\dt-implemetation\dt only.png" alt="RTLS digital twins diagram" /></p>
<h3 id="dt-ble-tag">DT BLE Tag</h3>
<p>This DT ‘reflects’ real-life object ‘ble tag’.</p>
<ul>
<li>Input stream – RSSI signals from BLE Anchors</li>
<li>Type – probabilistic estimator (particle filter).</li>
<li>Output – weighted mean estimation</li>
<li>Internal state – 3D location and on/off-site info</li>
</ul>
<h3 id="dt-employee">DT Employee</h3>
<p>This DT ‘reflects’ real-life object ‘employee’ that has smartphone (emulates BLE Tag for indoor positioning and periodically sends GPS location for outdoor positioning)</p>
<ul>
<li>2 input streams: 3D location estimation from ‘DT:BLE Tag’ and GPS signals</li>
<li>Type – aggregator – combines location information from sources of different periodicity into ‘global location’ info</li>
<li>Output - ‘global location’ info</li>
</ul>
<h2 id="example-2-video-processing">Example #2. Video processing</h2>
<p>Project in more details is described <a href="ml-from-monolith-to-micro-services.md">here</a></p>
<p>Overall simplified solution architecture</p>
<p><img src="assets\dt-implemetation\svkl archi.png" alt="Multiple cameras video processing IIoT" /></p>
<h3 id="digital-twins-only">Digital Twins only</h3>
<p><img src="assets\dt-implemetation\svkl archi dt only.png" alt="Multiple cameras video processing IIoT" /></p>Sergey KovalevDigital twins: implementation Let us dive deeper into Digital Twins implementation. The way I see 3 big classes of Digital Twins is described here. First, we will look at types of basic building blocks we need to implement Digital Twins in IIoT solution digital twin instance implementation extension management extension for digital twin instances data flowsExport costs from Azure2020-10-24T02:00:00+00:002020-10-24T02:00:00+00:00https://skaska.pro/azure-costs-export<p>Suppose your organization uses more than one cloud provider - Azure, AWS and Atlassian for different functions. Financial department desperately needs more advanced cost analysis by teams/projects/etc.
Requirement is simple - make costs data accessible in some corporate BI (PowerBI, Tableau). In this article will take a look at different aproaches for exporting Azure costs.
First resource you should get acquainted is <a href="https://azure.microsoft.com/en-us/services/cost-management/">Azure Cost Management</a></p>
<h1 id="manual-export">Manual export</h1>
<p><img src="\assets\azure-costs\manual export dataflow.png" alt="manual export" /></p>
<p>Export costs manually for last period from all subscriptions using <em>Cost management + Biling</em>.
Go to <em>Cost analysis</em> – set grouping options, periods and granularity.
Then go to <em>Download</em>, select <em>Excel</em>, press <em>Download Data</em> and you instantly get Excel file with data required.
Having real piece of data lets you start deeper conversation with financial department – what and how to import into BI sources. Some <em>tagging</em> and <em>resources</em> reorganization might be expected on Azure side.</p>
<h1 id="automatic-export">Automatic export</h1>
<p><img src="\assets\azure-costs\azure cost management scheduled export and import.png" alt="automatic export" /></p>
<p>Once requirements for data are specified in more details and everybody speaks the same language, export of Excel data might be automated.
Go to <em>Configuration:Exports</em> and create new data export with periodicity that you need.
Create Azure function import data into data source for BI and run it either periodically or on event based approach (which make more sense).
To add some resilience <em>data release</em> approach can be implemented (read more <a href="_posts\2020-10-16-data-release-management.md">here</a>).</p>
<h1 id="cost-management-api">Cost management API</h1>
<p><img src="\assets\azure-costs\azure cost management API.png" alt="Azure Cost Management API" /></p>
<p>If financial department is hungry for <em>rich daily data</em> - <a href="https://docs.microsoft.com/en-us/rest/api/cost-management/">Azure Cost Management REST API</a> goes to the rescue.
Architecture become a little more complicated, yet rather straight forward.
3 Azure functions connected via ServiceBus are used</p>
<ol>
<li>Scheduled function. Creates tasks for data extraction for every subscription in the account</li>
<li>Queries data from API and sends it to Bus in single chunk</li>
<li>Ingest data recieved from bus into storage (same function as in <em>Automatic Export</em>)</li>
</ol>
<h2 id="api-post-call-example">API POST call example</h2>
<h3 id="url">Url</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://management.azure.com/subscriptions/{subscription-id}/
providers/Microsoft.CostManagement/query?api-version=2019-11-01
</code></pre></div></div>
<p>Authentication is required</p>
<h3 id="request-body">Request body</h3>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Usage"</span><span class="p">,</span><span class="w">
</span><span class="nl">"timeframe"</span><span class="p">:</span><span class="w"> </span><span class="s2">"WeekToDate"</span><span class="p">,</span><span class="w">
</span><span class="nl">"dataset"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"granularity"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Daily"</span><span class="p">,</span><span class="w">
</span><span class="nl">"aggregation"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"totalCost"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"PreTaxCost"</span><span class="p">,</span><span class="w">
</span><span class="nl">"function"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Sum"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"grouping"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Dimension"</span><span class="p">,</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ResourceGroup"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Dimension"</span><span class="p">,</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ServiceName"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h3 id="sample-response">Sample response</h3>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"subscriptions/*****/providers/Microsoft.CostManagement/query/******"</span><span class="p">,</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"*****"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Microsoft.CostManagement/query"</span><span class="p">,</span><span class="w">
</span><span class="nl">"location"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"sku"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"eTag"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"nextLink"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"columns"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"PreTaxCost"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Number"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"UsageDate"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Number"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ResourceGroup"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ServiceName"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Currency"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"rows"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">[</span><span class="w">
</span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="mi">20201025</span><span class="p">,</span><span class="w">
</span><span class="s2">"*****"</span><span class="p">,</span><span class="w">
</span><span class="s2">"azure app service"</span><span class="p">,</span><span class="w">
</span><span class="s2">"RUB"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="p">[</span><span class="w">
</span><span class="mf">0.000075</span><span class="p">,</span><span class="w">
</span><span class="mi">20201025</span><span class="p">,</span><span class="w">
</span><span class="s2">"*****"</span><span class="p">,</span><span class="w">
</span><span class="s2">"bandwidth"</span><span class="p">,</span><span class="w">
</span><span class="s2">"RUB"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="err">.......</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>As you can see response is easy to undestand in ingest into storage.
You can play with API Query <a href="https://docs.microsoft.com/en-us/rest/api/cost-management/query/usage">on Microsoft documentation site</a></p>Sergey KovalevSuppose your organization uses more than one cloud provider - Azure, AWS and Atlassian for different functions. Financial department desperately needs more advanced cost analysis by teams/projects/etc. Requirement is simple - make costs data accessible in some corporate BI (PowerBI, Tableau). In this article will take a look at different aproaches for exporting Azure costs. First resource you should get acquainted is Azure Cost Management