hy8dZddlmZddlZddlZddlZddlZddlZddl Z ddl m Z m Z ddl m Z mZmZmZmZmZddlmZddlmZddlmZdd lmZmZdd lmZmZdd lm Z m!Z!m"Z"dd lm#Z$dd lm%Z&ddl'm(Z(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/m0Z0ddl1m2Z2m3Z3ddl1m4Z5ddl6m7Z7m8Z8ddl9m:Z:m;Z;mZ>ddl?m@Z@ddlAmBZBddlCmDZDddlEmFZFddlGmHZHddlImJZJmKZKmLZLddlMmNZOdd lPm#ZQddlPmRZRmSZSe rddlTZUejeWZXeegeeFe ffeeeYgefe eFfZZeegeeFe ffefZ[Gdde\Z]Gd d!eYZ^Gd"d#eYZ_ dM dNd$Z`dOd%ZaGd&d'eDZbdPd(Zc dQd)Zd dRd*Ze dSd+Zf dTd,Zg dUd-Zh dVd.Zi dWd/Zj dXd0Zk dYd1Zl dZd2Zmddddd3 d[d4Zndddd5 d\d6Zodd7 d]d8Zpdddd5 d^d9Zqdddd5 d\d:Zrdd7 d]d;Zs d_ d`d<ZtGd=d>eDd?@ZuejGdAdBZwdadCZxdbdDZydEZzdddFddd?ddG dcdHZ{dddFddd?ddG dcdIZ|dJZ}e}e|_e}jdKdLe{_y)dz>Utilities for running language models or Chains over datasets.) annotationsN)datetimetimezone) TYPE_CHECKINGAnyCallableOptionalUnioncast)warn_deprecated) Callbacks)BaseLanguageModel) BaseMessagemessages_from_dict) ChatResult LLMResult)RunnableRunnableConfigRunnableLambdaconfig)utils)EvaluatorCallbackHandlerwait_for_all_evaluators)LangChainTracer)Client) get_git_infoget_langchain_env_var_metadata)EvaluationResult RunEvaluator) run_evaluator) as_runnableis_traceable_function)DatasetDataTypeExampleRun TracerSession)LangSmithError) HTTPError) TypedDict)Chain)load_evaluator) EvaluatorTypePairwiseStringEvaluatorStringEvaluator) evaluation)name_generationprogressceZdZdZy)InputFormatErrorz(Raised when the input format is invalid.N)__name__ __module__ __qualname____doc__e/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain/smith/evaluation/runner_utils.pyr5r5Ks2r;r5c$eZdZdZ ddZddZy) TestResultz1A dictionary of the results of a single test run.c|j}|jDcgc]}|jds|dvr|}}|jdj |dScc}w)zReturn quantiles for the feedback scores. This method calculates and prints the quantiles for the feedback scores across all feedback keys. Returns: A DataFrame containing the quantiles for each feedback key. )inputs.outputs. reference>inputoutputall)include)axis) to_dataframecolumns startswithdescribedrop)selfdfcolto_drops r<get_aggregate_feedbackz!TestResult.get_aggregate_feedbackUss   zz ~~BC))   {{5{)..wQ.??  sA"c  ddl}g}g}|djD]e\}}|d}|jd} t | t r'| jD cic] \} } d| |  } } } n | i} nd| i} i|djD cic] \} } d | |  c} } | } d |vrUt |d t r:| j |d jD cic] \} } d | |  c} } n|d | d <| j i|Dcic]}d |j|jc}|jd |d|jdd|j| |j|h|j||S#t$r}d}t||d}~wwxYwcc} } wcc} } wcc} } wcc}w)z#Convert the results to a dataframe.rNzfPandas is required to convert the results to a dataframe. to install pandas, run `pip install pandas`.resultsfeedbackrDrArCr@rBz reference.z feedback.Errorexecution_timerun_id)errorrWrX)index) pandas ImportErroritemsget isinstancedictupdatekeyscoreappend DataFrame)rNpdemsgindicesrecords example_idresultrUoutput_kvrDrfs r<rIzTestResult.to_dataframejs * "&y/"7"7"9 ' Jj)Hjj*G'4(8? H1HQC.!+HH"G,06w0E0E0GH1WQC=!#HAf$f[148HH9? 9L9R9R9TUA:aS)1,U&,K%8AkN HH=EF155'*AGG3F#ZZ0&,-=&>$jj2   NN1  NN: &= '@r||G733S *@ c" )  *II V Gs/FF9F? G !G  F6#F11F6N)return pd.DataFrame)r6r7r8r9rRrIr:r;r<r>r>Rs;@ @*-4r;r>c,eZdZdZdfd ZddZxZS) EvalErrorz"Your architecture raised an error.c (t|dd|i|y)NrVr:)super__init__)rNrVkwargs __class__s r<rxzEvalError.__init__s /u//r;cT ||S#t$r}d|d}t||d}~wwxYw)Nz%'EvalError' object has no attribute '')KeyErrorAttributeError)rNnamergrhs r< __getattr__zEvalError.__getattr__s< -:  -9$qAC %1 , -s '"')rV BaseExceptionryrrrNone)rstrrrr)r6r7r8r9rxr __classcell__)rzs@r<rurus,0-r;rucN t|tr`|jj}|j7jjj}d|d|d|d}t |fdSt|t r|St|tr| fdSt|rt|rttt| fdS |}tt| t|t r|Sttt|rttt| fd St|ts fd S S|S#t$rNtt|}tj|}t j#d|t%| fd cYSwxYw) zForgive the user if they pass in a chain without memory instead of a chain factory. It's a common mistake. Raise a more helpful error message as well.a$Cannot directly evaluate a chain with stateful memory. To evaluate this chain, pass in a chain constructor that initializes fresh memory each time it is called. This will safegaurd against information leakage between dataset examples. For example: def chain_constructor(): new_memory = z(...) return z*(memory=new_memory, ...) run_on_dataset("z", chain_constructor, ...)cSNr:)chainsr<z(_wrap_in_chain_factory..sur;cSrr:)lcfsr<rz(_wrap_in_chain_factory..ssr;cSrr: runnable_sr<rz(_wrap_in_chain_factory..9r;z'Wrapping function %s as RunnableLambda.cSrr:)wrappedsr<rz(_wrap_in_chain_factory..s7r;cSrr:rsr<rz(_wrap_in_chain_factory..rr;ctSr)r) constructorsr<rz(_wrap_in_chain_factory..s >+6r;)r_r,rzr6memory ValueErrorrrcallabler#r"r r TypeErrorinspect signatureloggerinfor) llm_or_chain_factory dataset_name chain_class memory_classrh_model user_funcsigrrrrrs @@@@@r<_wrap_in_chain_factoryrs &.$oo..  & & 2 <<11::L$%1>2)]##/.0J L S/ !&(9:##&1"$% !5 6#D3G$HII$ $ #)+F8%9: f/ 0M h!7 8#D6$:;I$ $&(+6 6 ) #X';z_get_prompt..s> #$Jq# > z,Expected list of strings for 'prompts', got rGc3<K|]}t|tywrrrs r<rz_get_prompt..s.Saz!S/A.Srz)LLM Run expects string prompt input. Got z5LLM Run expects 'prompt' or 'prompts' in inputs. Got rz)LLM Run expects single prompt input. Got z prompts.) r5r_rtyper6listrElennextitervalues)inputsrhrprompt_s r< _get_promptrs +s##G6&*C06tF8 (.y(9> ; VI./889; #3' '# V tFMMO,- gs #iG  &3.S7.S+SG=fXFC"3' 'EfXNs## 7|qqz 5c'l^9 MC 3 r;ceZdZUdZded<y)ChatModelInputzVInput for a chat model. Parameters: messages: List of chat messages. zlist[BaseMessage]messagesNr6r7r8r9__annotations__r:r;r<rrs  r;rc|s d}t||j}d|vr|jd|d<n3t|dk(r%t t |j |d<d|vrX|d}t|trtd|Dr|g}t|dk(rt|d|d<|Sd}t|d|}t|) zGet Chat Messages from inputs. Args: inputs: The input dictionary. Returns: A list of chat messages. Raises: InputFormatError: If the input format is invalid. rrrCrGc3<K|]}t|tywr)r_r`rs r<rz _get_messages..5s2 $%Jq$ 2 rrzGBatch messages not supported. Please provide a single list of messages.zMChat Run expects single List[dict] or List[List[dict]] 'messages' input. Got ) r5copypoprrrrr_rrEr)rrh input_copy raw_messagess r< _get_messagesr s +s##JV(nnZ8 7 V "4 #89 7*!'* lD )c2 )52 / )>L |  !"4\!_"EJw  , #3' ' h  3 r;c|rg||jxsi}t|ts@t|trt d|Dsd|dt |d}t |yy t|jxsiy#t $rJ t|jxsiYy#t $r!}d|jd}t ||d}~wwxYwwxYw)Nc3<K|]}t|tywrr_rrrhs r<rz>_validate_example_inputs_for_language_model..RsISJsK0IrzWhen using an input_mapper to prepare dataset example inputs for an LLM or chat model, the output must a single string or a list of chat messages. Got: of type .zvExample inputs do not match language model input format. Expected a dictionary with messages or a single prompt. Got: z Please update your dataset OR provide an input_mapper to convert the example.inputs to a compatible format for the llm or chat model you wish to evaluate.) rr_rrrErr5rr) first_example input_mapper prompt_inputrherr2s r<+_validate_example_inputs_for_language_modelrJs#M$8$8$>B? ,, |T *ILII'yl1C0DAG  #3' 'J- 6  ,,2 3 6 6m228b9# 6*112GG's+5 6 6s*+B CB** C3CCCc:|r||jxsi}t|jj|}t |t sd|dt |d}t||r+d|jd|j}t|y|j}t|jj|}t|dk(rt|jdk(ry|r+d|jd|j}t|y) zB? 5++,77 E ,-&yl1C0DAG  #3' ' $//0 8I8I8K7LN #3' ' %++ 5++,77 E |  !c%*:*:&;q&@  #--./$))+, . #3' 'r;ct|tr t||y|}t|trt |||yt|t rt jd|yy)z9Validate that the example inputs are valid for the model.z Skipping input validation for %sN)r_rrr,rrrdebug)examplerrrs r<_validate_example_inputsrsV &(9:3G\J$& eU # .w| L x ( LL;U C)r;c :|rt|trd\}}d}nEd}|}t|tr |jnd}t|tr |jnd}t ||||dj rt|dj nd||}|Sd}|S)z&0 #J HH(*E-7u-E))4J/9%/G%++TK-   )1!)<)< NN8    [)Q.$Q   S%5%9 ;   r;c|jr'|j}|r||vrd|d|}t||S|r$t|dk(rtt |}|Sd}|S)NzReference key z! not in Dataset example outputs: rG) reference_keyrrrr)rexample_outputsrrhs r<_determine_reference_keyrs,, }OC 0%%4$57 S/ !  S1Q6T/23   r;c ~t|tr|St|ttfr5t|ts t|}t ||}|j } nt|t jrd|i|j} t |jfi| }|jj } t|t jr`|jxs|}|jxs|}|jxs|}n/t|r t|Sdt!|} t#| t|t$rK|j&r|d| d|d} t#| t(j*j-||||||| g} | St|t.rd| d } t1| d| d } t1| ) N)rrzUnknown evaluator type: zPMust specify reference_key in smith_eval.RunEvalConfig to use evaluator of type z) with dataset with multiple output keys: r)rrrtagszRun evaluator for z is not implemented. PairwiseStringEvaluators compare the outputs of two different models rather than the output of a single model. Did you mean to use a StringEvaluator instead? See: https://python.langchain.com/docs/guides/evaluation/string/z is not implemented)r_r r.rr-valuesmith_eval_config EvalConfig get_kwargsevaluator_typeSingleKeyEvalConfigrrrrrun_evaluator_decrrr0requires_reference smith_evalStringRunEvaluatorChainfrom_run_and_data_typer/NotImplementedError) eval_configeval_llmrrrrrr evaluator_ eval_type_tagryrhr!s r<_construct_run_evaluatorrs+|,+ s34+}5' 4K#KX> #)) K!2!=!= >>[%;%;%=>#K$>$>I&I #2288 k#4#H#H I#--:I(77I>N'55FM +  --(k):(;<o*o.  ( (]-B&&3_577F6GqJ  S/ !"::QQ   )'R ,  J 7 8 0Q Q "#&&#=/1DE!#&&r;cTt||}t||}t||}|||fSr)rrr)rrrrrrrs r< _get_keysrTs5 %VZ8I.v{CN,V_EM nm 33r;c g}d\}}} |js(|jr.td|jDrt||||\}}} |jD]/} t | |j |||| ||} |j | 1|jxsg} | D]} t| tr|j | %t| tr5|j tjj| ||||| jt| r|j t| d| d}t||S)z Load run evaluators from a configuration. Args: config: Configuration for the run evaluators. Returns: A list of run evaluators. NNNc3<K|]}t|tywr)r_r0)rrgs r<rz'_load_run_evaluators..usQ1 1o.Qr)rrrzUnsupported custom evaluator: z+. Expected RunEvaluator or StringEvaluator.) evaluatorscustom_evaluatorsanyrrrrdr_r r0rrrrrr)rrrrrrrrrrrr!r custom_evaluatorrhs r<rr`sv"N/?,I~}   Q8P8PQ Q3<     4 0 >= (( - 0  OO         m, -006B-" & 5  ! !"2 3 (/ :  ! !22II$'#1"/ J & '  ! !"34D"E F11A0BC=> S/ !+". r;r callbacksrmetadatac  K|z||}t|ts"t|trAtd|Dr/|j |t ||xsg|xsid{Sd|d}t | t|}|j |t ||xsg|xsid{} | S7R7#t $rBt|} |jdi| dt ||xsg|xsiid{7} Y| SwxYww) aAsynchronously run the language model. Args: llm: The language model to run. inputs: The input dictionary. tags: Optional tags to add to the run. callbacks: Optional callbacks to use during the run. input_mapper: Optional function to map inputs to the expected format. Returns: The LLMResult or ChatResult. Raises: ValueError: If the LLM type is unsupported. InputFormatError: If the input format is invalid. Nc3<K|]}t|tywrrrs r<rz_arun_llm..OSJsK0Orrrrrz%Input mapper returned invalid format 3 Expected a single string or list of chat messages.rr:) r_rrrEainvokerr5rr) rrrrrrprompt_or_messagesrhr llm_output llm_inputss r< _arun_llmrsU0)&1 (# . )4 0O6C:D>CAD DD D D  Drrrc K||n||}t|tr}t|trmt|dk(r_|jrSt t |j}|j|t||xsg|xsid{}|St|xsg||xsi} |j|| d{}|S797w)z%Run a chain asynchronously on inputs.NrGrrrrr) r_r,r`rrrrrrr rrrrrrinputs_valrDrunnable_configs r< _arun_chainr$s%,f,v2FG5% w % LA    4()*}} !#ZR!R%   M )^  }}W_}EE M Fs$BCC2CC CC)rc Kt|trdnd}d} t|tr=t||jxsi|d|d||j dd{}nC|}t ||jxsi|d|d||j dd{}|}|S7L7 #t $rD}tjd||j|j|t| }Yd}~|Sd}~wwxYww) aAsynchronously run the Chain or language model. Args: example: The example to run. llm_or_chain_factory: The Chain or language model constructor to run. tags: Optional tags to add to the run. callbacks: Optional callbacks to use during the run. input_mapper: Optional function to map the input to the expected format. Returns: A list of outputs. LLMr,Nrrrrz*%s failed for example %s with inputs %s %srV) r_rrrr^r$ Exceptionrridru) rrrr chain_or_llmrlrDrrgs r<_arun_llm_or_chainr+s(02CD'F$ *,= > )$$"F^ -)J/ !F)*E&$"F^ -)J/ F M9 $ 9  JJ NN  # M$sTDAB2!B."AB2%B0&B2,D.B20B22 C?;9C:4D:C??Dc |t||}t|ts"t|tr;td|Dr)|j |t ||xsg|xsi}|Sd|d}t | t|} |j | t ||xsg|xsi}|S#t $r4t|} |jdi| dt ||xsii}Y|SwxYw) a Run the language model on the example. Args: llm: The language model to run. inputs: The input dictionary. callbacks: The callbacks to use during the run. tags: Optional tags to add to the run. input_mapper: function to map to the inputs dictionary from an Example Returns: The LLMResult or ChatResult. Raises: ValueError: If the LLM type is unsupported. InputFormatError: If the input format is invalid. c3<K|]}t|tywrrrs r<rz_run_llm..grrrrz'Input mapper returned invalid format: rr)rrr:) r_rrrEinvokerr5rr) rrrrrrrrrh llm_promptsrs r<_run_llmr0Js-2)&1 (# . )4 0O Mr;c t|trdnd}d} t|tr5t||jxsi|d|d||j d}n;|}t ||jxsi|d|d||j d}|}|S#t $rZ}t|j} tjd||j|j| |t| }Yd}~|Sd}~wwxYw) a Run the Chain or language model synchronously. Args: example: The example to run. llm_or_chain_factory: The Chain or language model constructor to run. tags: Optional tags to add to the run. callbacks: Optional callbacks to use during the run. Returns: Union[List[dict], List[str], List[LLMResult], List[ChatResult]]: The outputs of the model or chain. r&r,NrrrrzC%s failed for example %s with inputs %s Error Type: %s, Message: %sr') r_rr0rr^r2r(rr6rrr)ru) rrrrr*rlrDrrg error_types r<_run_llm_or_chainr5s*02CD'F$ *,= >"$$"{#F^)J/ F)*E$"{#F^)J/ F M  $!W%%  R  JJ NN   # M $sBB C?%AC::C?c t||}|j|}t|j|j|} | sd|d} t | | D cgc]} | j s| j } } | r t| nd} | r| jnd} |xsi}t}|ri|d|i}||d<|j||j|rd|ini|}|j d|jz}t#d|d|d|d|j d|||| fScc} w#tt tf$rG}d t|vrtj}d |d |d |d }d|d|} t | |d}~wwxYw)N)r) dataset_idas_ofzDataset z has no example rows.gitdataset_versionr)reference_dataset_id project_extrarzalready exists z+ run_on_dataset( ... project_name="z - z", # Update since z already exists ) z Test project z/ already exists. Please use a different name: z/compare?selectedSessions=z)View the evaluation results for project 'z' at: z View all tests for Dataset z at: T)flush)r read_datasetr list_examplesr)r modified_atmax isoformatrcreate_projectr*r)ruuiduuid4urlprint)clientrr project_nameproject_metadatarr: wrapped_modeldatasetrrhexr@max_modified_atinferred_versiongit_infoprojectrguid example_msgcomparison_urls r<_prepare_eval_runrUs++?NM!!|!>GKG+6c+&4O6E0024%+1r>  " x   /?*+'' !(,064.b% ( *[[%? |#LLN 3L>B !&&2^6'++ H  '7H 44YH, z> 2% CF * jjl .C5(:<.I L>*- ! o1$%s&D+.D+"AD00F AFF c0eZdZUdZded<ded<ded<y) _RowResultz5A dictionary of the results for a single example row.z Optional[list[EvaluationResult]]rUzOptional[float]rW Optional[str]rXNrr:r;r<rWrW*s?..## r;rWF)totalceZdZUdZded<ded<ded<ded <d ed <d Zd ed< ddZddZddZ ddZ d ddZ e d ddZ y )_DatasetRunContainerz3A container to help manage the state of a eval run.rrHr(rQMCFrK list[Example]rzlist[RunnableConfig]configsNz6Optional[list[smith_eval_config.BATCH_EVALUATOR_LIKE]]batch_evaluatorsc `i}t|j|D]\}}tt|j t |j i}|j|j dg|j d|j dd|t |j <t|tr&|j|t |j d<n||t |j d<|js|j|t |j d<|S)NrUrWrX)rCrUrWrXrVrDrB) ziprr rWr^rr)rr_rurVr)rN batch_resultsall_eval_resultsrTrrD row_results r<_merge_test_outputsz(_DatasetRunContainer._merge_test_outputs=s "4==-@ HOGVj*:*>*>s7::PR*STJ &NN:r:",..1A"B$..2 (GC O $ &),4:LLGJJ(15;GJJ(28?GJJ(5 Hr;c |j}|sgS|jDcgc]}|t|j}}g}tj j 5}|D]} |||j}t|tr|j}|jtt||j|jjfi|d|jjd ddd|Scc}w#t $r"t"j%dt'|YwxYw#1swY|SxYw)N)rX project_idz Error running batch evaluator %s)r_rrr) concurrentfuturesThreadPoolExecutorr_rr`rdr submitrHcreate_feedbackrQr(r exceptionrepr) rNrunsr r runs_listaggregate_feedbackexecutor evaluatorrls r<_run_batch_evaluatorsz*_DatasetRunContainer._run_batch_evaluatorsSs(** I:>--HwT#gjj/*H H    2 2 4 '  &y$--@F!&*:;!'&--d4.@A#HOO 33  $#'<}tt|dD]$}t|trT|j }|j D]4\\}}}|jt|ijd|i6ht|tsy|j} | r3| jr'| j| jz jnd} | rt| jnd} |jt|j ij| | | d| |t|j <'Att"tt$f||fS)NrrU)rWrXrun)r^r rr_rlogged_eval_resultsr] setdefaultrrar latest_runend_time start_time total_secondsr)rkr`rW) rNrcall_runsccallback eval_results_rkrorvrWrXs r<_collect_metricsz%_DatasetRunContainer._collect_metricslsX!# =A q~6 =h(@A#+#?#?L.:.@.@.B*J(33C ORHOO'O /:"--C3<<6EEG!# -0S[TF$//H4G4G0H"MTT.<&,#&:=HS!4!456- = =0Dj)+;rQr)rNrbrcr}rqrTs r<_collect_test_resultsz*_DatasetRunContainer._collect_test_resultss  9:!%)%:%:%<"(!  KK5 6!%!;!;H!E **=:JK**0  r;c|j|}|r |j}t| |j j|jjtjtj|S#t$r"}tj d|dYd}~{d}~wwxYw#t$r#}tj d|dYd}~|Sd}~wwxYw)Nz&Failed to print aggregate feedback: %sT)exc_info)rzzFailed to close project: %s)rrR_display_aggregate_resultsr(rrrHupdate_projectrQr)rnowrutc)rNrbverboserT agg_feedbackrgs r<finishz_DatasetRunContainer.finishs ,,];  Y&==? *<8 J KK & & !hll3 '   Y EqSW XX Y J LL6DL I I Js/BAB3 B0B++B03 C<CCc ,|xstj}| r| si} | jd| it||||| || \} } }}|xsg}| jj dxsij D]\}}|jd|d|d| jdi}| r| |d<t|} t| |||jxstj}t|d| |tjt!|}|Dcgc]O}t#t%| j&||j(t+|xsg||j(d |g||| Q}}||| | |||r|j, Sd Scc}w) N revision_id)rJrr:r9zgit:=r:r)rIrHrk)r rHrkmax_concurrency)rrrr)rHrQrKrr^r_)r2 random_namerarUrr^r]rdrrrr%kvrr3ProgressBarCallbackrrrrr)rr_)clsrHrrrIr1rrconcurrency_levelrJrr:rKrQrLrrnro run_metadatar progress_barrr^s r<preparez_DatasetRunContainer.prepares$D'B'B'D ##%  # #]K$@ A4E   -+5 1 wzr%%))%06B==? (DAq KK$qc1# ' ()7+;+;)F)rbrrboolrrr>)NNNNNN)rHrrrrMODEL_OR_CHAIN_FACTORYrIrXr1"Optional[smith_eval.RunEvalConfig]rOptional[list[str]]rOptional[Callable[[dict], Any]]rintrJOptional[dict[str, Any]]rrXr:Optional[Union[datetime, str]]rrr[) r6r7r8r9rr_rertrrr classmethodrr:r;r<r[r[2sF= N  !!OSLS0  ,"2G: D   *  ,:>$(8H H H 5 H $ H 7 H "H 6H H 3H #H 8H  H H r;r[c~ ddlm}|}|duxrdtt|vS#t$rYywxYw)Nr) get_ipythonzmqshellF)IPython.core.getipythonrrrr\)rress r<_is_jupyter_environmentrsB7m}D(IZ3tCy>-II s -0 <<ctrddlm}m}||d||y|j dd}t dt |y)Nr)HTMLdisplayz

Experiment Results:

c |dS)Nz.2fr:)xs r<rz,_display_aggregate_results..s aWr;right) float_formatjustifyz Experiment Results:)rIPython.displayrr to_stringrG)aggregate_resultsrrformatted_strings r<rrsR 1345!",66-7  &' r;aThe input_mapper argument is deprecated and will be removed in a future release. Please add a RunnableLambda to your chain to map inputs to the expected format instead. Example: def construct_chain(): my_chain = ... input_mapper = {'other_key': 'MyOtherInput', 'my_input_key': x} return input_mapper | my_chain run_on_dataset(..., llm_or_chain_factory=construct_chain) (See https://api.python.langchain.com/en/latest/schema/langchain.schema.runnable.base.RunnableLambda.html)r)r1r:rrIrJrrc K| jdd} | rtdtd| tj d} | jdd} | rtddd| r tdd | j d d |xs t }tj|||||| | ||| | } tj| jd j dgttjt| j | | j"| jd{}| j%||S7w)Nr0.0.305Tmessagependingrr0.1.9qThe tags argument is deprecated and will be removed in a future release. Please specify project_metadata instead.PThe following arguments are deprecated and will be removed in a future release: rrremovalrJrr:rrrrr)rr _INPUT_MAPPER_DEP_WARNINGrr^rrr[rrunnable_utilsgather_with_concurrencyr^map functoolspartialr+rKrr)rHrrr1r:rrIrJrrryrr containerrbs r<arun_on_datasetr'sd::nd3L +DdS46::=I ::fd #D  U   4{{}oQ    vxF$,, )'- I)@@!  !23    "%.%<%<)          M   M7  ;; sD,E.E/Ec N| jdd} | rtdtd| jdd} | rtddd| tj d} | r tdd | j d d |xs t }tj|||||| | ||| | } |d k(rJt| j| jDcgc]\}}t||| j|  }}}ntj| jd 5}t!|j#t%j&t| j| | j| j}ddd| j)|Scc}}w#1swY"xYw)NrrTrrrrrrrrrrrr)rr rrr^rrr[rrarr^r5rKr#get_executor_for_configrrrrr)rHrrr1r:rrIrJrrryrrrrrrbrrs r<run_on_datasetrhs::nd3L +DdS ::fd #D  U  46::=I   4{{}oQ    vxF$,, )'- IA$'y'9'99;L;L#M   %.%<%<)     4 4Y5F5Fq5I J h  %%)-6-D-D%1 &&%% M    M7  ;;/   s#F%AFF$a1 Run the Chain or language model on a dataset and store traces to the specified project name. Args: dataset_name: Name of the dataset to run the chain on. llm_or_chain_factory: Language model or Chain constructor to run over the dataset. The Chain constructor is used to permit independent calls on each example without carrying over state. evaluation: Configuration for evaluators to run on the results of the chain concurrency_level: The number of async tasks to run concurrently. project_name: Name of the project to store the traces in. Defaults to {dataset_name}-{chain class name}-{datetime}. project_metadata: Optional metadata to add to the project. Useful for storing information the test variant. (prompt version, model version, etc.) client: LangSmith client to use to access the dataset and to log feedback and run traces. verbose: Whether to print progress. tags: Tags to add to each run in the project. revision_id: Optional revision identifier to assign this test run to track the performance of different versions of your system. Returns: A dictionary containing the run's project name and the resulting model outputs. For the (usually faster) async version of this function, see :func:`arun_on_dataset`. Examples -------- .. code-block:: python from langsmith import Client from langchain_openai import ChatOpenAI from langchain.chains import LLMChain from langchain.smith import smith_eval.RunEvalConfig, run_on_dataset # Chains may have memory. Passing in a constructor function lets the # evaluation framework avoid cross-contamination between runs. def construct_chain(): llm = ChatOpenAI(temperature=0) chain = LLMChain.from_string( llm, "What's the answer to {your_input_key}" ) return chain # Load off-the-shelf evaluators via config or the EvaluatorType (string or enum) evaluation_config = smith_eval.RunEvalConfig( evaluators=[ "qa", # "Correctness" against a reference answer "embedding_distance", smith_eval.RunEvalConfig.Criteria("helpfulness"), smith_eval.RunEvalConfig.Criteria({ "fifth-grader-score": "Do you have to be smarter than a fifth grader to answer this question?" }), ] ) client = Client() run_on_dataset( client, dataset_name="", llm_or_chain_factory=construct_chain, evaluation=evaluation_config, ) You can also create custom evaluators by subclassing the :class:`StringEvaluator ` or LangSmith's `RunEvaluator` classes. .. code-block:: python from typing import Optional from langchain.evaluation import StringEvaluator class MyStringEvaluator(StringEvaluator): @property def requires_input(self) -> bool: return False @property def requires_reference(self) -> bool: return True @property def evaluation_name(self) -> str: return "exact_match" def _evaluate_strings(self, prediction, reference=None, input=None, **kwargs) -> dict: return {"score": prediction == reference} evaluation_config = smith_eval.RunEvalConfig( custom_evaluators = [MyStringEvaluator()], ) run_on_dataset( client, dataset_name="", llm_or_chain_factory=construct_chain, evaluation=evaluation_config, ) zrun_on_dataset(zawait arun_on_dataset()z )rrrrrrr\)rdict[str, Any]rrr)rrrrr`)rr&rrrrr)rr&rr,rrrrr)rr&rr\rrrrr) rr\rr]r1rrr%rrzOptional[list[RunEvaluator]])rsmith_eval.RunEvalConfigrrrrrX)rrrrrrrX)rrrrrrrX)rzYUnion[smith_eval_config.SINGLE_EVAL_CONFIG_TYPE, smith_eval_config.CUSTOM_EVALUATOR_TYPE]rzOptional[BaseLanguageModel]rrrr%rrrrXrrXrrXrrr ) rrrrrrrrrrz2tuple[Optional[str], Optional[str], Optional[str]])rrrrrr%rrrrrrrrzlist[RunEvaluator])rrrrrrrr rrrrrrUnion[str, BaseMessage])rzUnion[Chain, Runnable]rrrr rrrrrrrrzUnion[dict, str]) rr&rrrr\rrrrz'Union[dict, str, LLMResult, ChatResult])rrrrrr rrrrrrrrrr )rHrrrrrrIrrJrrrr:zOptional[Union[str, datetime]]rrz1tuple[MCF, TracerSession, Dataset, list[Example]])rrr)rrsrrr)rHzOptional[Client]rrrrr1rr:rrrrIrXrJrrrrrXryrrrr)r9 __future__rconcurrent.futuresrh dataclassesrrloggingrDrrtypingrrrr r r langchain_core._apir langchain_core.callbacksr langchain_core.language_modelsrlangchain_core.messagesrrlangchain_core.outputsrrlangchain_core.runnablesrrrrr#rr!langchain_core.tracers.evaluationrr langchain_core.tracers.langchainrlangsmith.clientr langsmith.envrrlangsmith.evaluationrr r!rlangsmith.run_helpersr"r#langsmith.schemasr$r%r&r'r(langsmith.utilsr)requestsr*typing_extensionsr+langchain.chains.baser,langchain.evaluation.loadingr-langchain.evaluation.schemar.r/r0langchain.smithr1rlangchain.smith.evaluationrr2r3r[rf getLoggerr6rr`rr\r(r5r>rurrrrrrrrrrrrrrrr$r+r0r2r5rUrW dataclassr[rrrrr_RUN_ON_DATASET_DOCSTRINGreplacer:r;r<rs+D" '0.<C8MM><=#FELL*''7 5B@   8 $ Ruh' '( dVS[    HRuh// 02C CD3y3E4E4P - - ': 0: :  : z. b Y & T 6 61 6  6F%(%( %(2%( %(PD DD2D D(3  " < $#6 $$4 $(&BB * B  BB)B!BB"BBJ 4 $ 4# 4% 4) 4 8 4A $AAA) A $ A % AAV!%48)-A A A  A  A 2 A'AAR!%48)-! !! !!  ! 2 !'!!R59 5 5 5 5 2 5 - 5@!%48)-? ? ??  ? 2 ?'??N!%48)-! !! !!  ! 2 !'!!R59 8 8 8 8 2 8 - 8@26 $6:<5 <5<51<5 <5 / <5  <54<57<5~% G G G T  :,6:6:"&15!%>< ><><1>< 3 >< 4 ><>< ></><><><><>